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I .  INTRODUCTION 


A.  BACKGROUND 

Soviet  military  retrenchment  and  political  collapse  have 
significantly  affected  U.S.  strategic  and  tactical  force 
structure  and  requirements.  United  States  national  defense 
goals  and  requirements  presently  lack  the  refinement  and 
broad  consensus  forty  years  of  Soviet  confrontation 
engendered.  The  Federal  budget  deficit  is  viewed  as  a 
severe  threat  to  the  health  and  future  of  the  nation's 
economy  and  the  Department  of  Defense  (DoD)  budget  will 
remain  under  severe  pressure  throughout  the  1990s.  Not  only 
will  there  be  less  money  to  support  existing  programs  and 
initiatives,  but  the  Department's  ability  to  introduce  new 
systems  will  be  increasingly  constrained. 

The  systems  purchased  to  support  the  changes  in  military 
doctrine  that  an  altered  international  landscape  requires 
must  have  greater  flexibility,  reliability  and 
interoperability.  Simultaneously,  these  systems  must  be 
brought  on-line  without  the  schedule  and  cost  over-runs  that 
plagued  DoD  during  the  years  of  sustained  Soviet  threat. 
Acquisition  programs  that  experience  perturbations  in  terms 
of  schedule,  cost,  or  performance  will  not  only  fail  to 
mature  but  will  cause  damaging  ripple  effects  to  other 
programs  because  of  the  smaller  total  force  structure. 
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For  the  last  forty  years  the  United  States  has  relied 
upon  an  acquisition  strategy  of  fielding  technologically 
superior  systems  to  counter-balance  the  numerical  advantages 
of  a  perceived  Soviet  threat.  Technical  sophistication  has 
permeated  all  levels  of  force  structure  and  doctrine, 
requiring  state-of-the-art  performance  from  individual 
hardware  components.  Increasingly  complex  data  acquisition, 
processing,  and  exchange  networks  are  required  to  support 
the  command  and  control  requirements  of  forces  equipped  with 
these  systems . 

Despite  continuous  efforts,  the  United  States  has  been 
unable  to  consistently  develop,  field,  and  maintain  a  force 
structure  that  possesses  an  effective  technological 
advantage  over  Warsaw  Pact/Soviet  forces  [Ref.  1].  This  has 
forced  the  U.S.  defense  establishment  to  continually 
attempt  the  development  and  fielding  of  ever  more  advanced 
capabilities  subject  to  increasingly  rigid  cost  and  time 
constraints.  Aggressive  performance  and  schedule  goals 
sometimes  have  been  specified  without  a  consistent  or 
coherent  assessment  of  the  risks. 

While  much  criticism  has  been  leveled  at  DoD  because  of 
past  cost  and  schedule  over  runs  and  performance  failures 
[Ref.  2:pp.  xxiii-xxv],  there  was  often  sufficient  breadth 
in  new  capability  development  efforts  and  elasticity  in 
budgets  to  ameliorate  the  effects  of  performance 
short-falls,  cost,  and  schedule  over-runs  in  any  one 
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particular  program.  The  Navy  was  able  to  field  improvements 
to  Anti-Air  Warfare  (AAW)  capabilities  with  three  systems 
through  the  1970s  and  1980s.  The  AEGIS  ship-board  radar  and 
missile  system,  F-14  Tome at /PHOENIX  missile  system,  and  the 
F/A-18  Hornet,  were  eventually  fielded  despite  cost, 
schedule,  and  performance  problems  with  all  three.  In  the 
future,  programs  which  experience  these  problems  could  have 
a  devastating  impact  on  force  structures  and  capabilities. 

The  Navy's  A-12  Avenger  program,  employing  "stealth" 
technology,  was  designed  as  a  replacement  for  the  venerable 
A-6  Intruder  medium  attack  aircraft.  The  A-12  enjoyed  the 
highest  priority  within  the  service.  An  A-6  Upgrade  program 
{A-6F)  was  cancelled  so  that  resources  available  for  the  A- 
12  program  would  not  be  constrained.  Cancellation  of  the  A- 
12  program  not  only  resulted  in  the  significant  delay  of  a 
vitally  needed  capability,  but  has  seriously  weakened  the 
Navy's  ability  to  fund  other  important  aviation  programs  in 
a  climate  of  fiscal  austerity.  Financial  resources  to  meet 
heavy  lift,  maritime  patrol,  AAW,  and  carrier-based  early 
warning  programs  are  all  jeopardized  [Refs.  3,  4]. 

The  current  acquisition  environment  is  characterized  by 
concurrent  development  of  systems  that  will  operate  with  a 
wide  variety  of  forces  in  numerous  environments.  Single¬ 
service  programs  tailored  to  meet  unique  service  objectives 
will  receive  far  greater  scrutiny  and  will  become  rarities. 
The  impact  of  a  termination  could  have  severe  effects  on  the 
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capabilities  of  the  entire  DoD  force  structure  in  this  sort 
of  environment.  Termination  of  the  A~12  has  stopped  Air 
Force  efforts  to  develop  a  derivative  A-12  as  a  replacement 
for  the  F-15  Eagle  and  F-111  [Ref.  3]. 

B.  OBJECTIVE 

The  objective  of  this  thesis  is  to  examine  existing 
acquisition  program  risk  assessment  theories  and  methods, 
evaluate  their  efficacy  as  employed  in  commercial  settings, 
and  look  for  ways  that  successful  commercial  approaches 
could  be  adapted  for  use  within  the  unique  DoD  acquisition 
decision  environment.  Specifically,  this  research  attempts 
to  create  a  useful  and  flexible  definition  of  technological 
risk  and  to  identify  risk  assessment  methods  that  are 
accessible  to  DoD  program  managers  and  their  staffs. 

C.  RESEARCH  QUESTIONS 

Commencing  with  the  axiomatic  framework  constructed  by 
von  Neumann  and  Morgenstern  [Ref.  5],  a  large  body  of  work 
has  addressed  risk  assessment  for  individual  or  corporate 
decisions.  Transference  of  these  theories  and  techniques  to 
the  DoD  acquisition  setting  requires  investigation. 
Assumptions  about  the  nature  and  availability  of 
alternatives  for  subsequent  "risk  premium"  and  certainty 
equivalent  analysis  may  not  directly  apply  to  the  DoD 
environment.  This  thesis  will  examine  the  following  aspects 
of  risk  assessment  from  a  Decision  Theory  perspective: 


4 


1.  What  are  the  underlying  theories  and  methodology 
supporting  risk  assessment  in  decision  analysis? 

2.  How  are  these  methods  applied  in  commercial 
settings?  What  are  their  strengths  and  weaknesses? 

3.  What  are  the  major  factors  defining  the  DoD 
technical  risk  assessment  environment?  Are  these  factors 
similar  to  those  found  in  the  commercial  world? 

4.  Evaluate  whether  or  not  the  techniques  used  by 
commercial  acquisition  program  managers  can  be  applied 
directly  to  the  DoD  acquisition  case.  If  not,  what  changes 
would  be  required. 

5.  Suggest  possible  improvements  to  existing  DoD 
technical  risk  acquisition  methods. 

This  thesis  will  not  examine  cost  and/or  schedule  issues 
nor  the  procedures  for  determining  the  performance 
requirements  specified  for  new  systems. 

D.  SCOPE  AND  LIMITATIONS 
1.  Scope 

There  are  two  basic  approaches  used  to  define  risk 
assessment:  normative  or  prescriptive,  and  descriptive.  The 
normative  approach  strives  to  define  ways  in  which  a 
"rational"  person  should  go  about  confronting  the  Universe. 
The  descriptive  approach  primarily  seeks  to  understand  how 
real  people  make  real  decisions. 
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There  exists  a  resultant  tension  between  these  two 
approaches  that  must  be  monitored  if  usable  decision  tools 
are  to  be  fashioned.  However,  for  the  purposes  of  this 
thesis,  the  emphasis  will  be  on  how  a  normative  approach  can 
be  applied  to  the  risk  assessment  problem  of  the  program 
manager  or  acquisition  executive. 

2.  Limitations 

Within  the  normative  scheme  there  are  three 
decision-making  approaches,  all  using  some  form  of  divide 
and  conquer” : 

1.  Cost  Benefit  Theory  (CBT) 

2.  Social  Welfare  Theory  (SWT) 

3.  Decision  Theory  (DT) 

All  three  use  a  standard  partitioning  of  the  decision-making 
problem,  seeking  to  counter-pose  knowledge  and  values  at 
each  step  of  the  process  and  synthesize  these  in  a  logically 

consistent  manner  [Ref.  6]. 

The  CBT  approach  is  basically  an  impersonal  one, 
based  upon  ^'scientific  objectivity.”  SWT  tends  to  focus  on 
the  organizational  dynamics  and  social  processes.  Decision 
Theory  frames  the  problem  within  the  context  of  the 
individual  as  the  decision-making  entity. 

While  subjective  in  its  world  view.  Decision  Theory 
seems  suitable  for  helping  individual  acquisition  decision 
makers  confronted  with  the  problem  of  technical  risk 
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assessment.  This  is  particularly  beneficial,  since  a  body 
of  objective,  verifiable  data  is  not  always  available  at  the 
early  stages  of  new  acquisition  projects  [Ref.  7]. 

E.  LITERATURE  REVIEW 

One  particular  aspect  of  the  DoD  acquisition  conundrum 
is  the  assessment  of  technical  risk.  A  1986  GAO  Report, 
commissioned  by  the  U.S.  Senate  Committee  on  Governmental 
Affairs,  found  numerous  inconsistencies  and  omissions  in  how 
acquisition  program  offices  tackled  the  problem  of  technical 
risk  assessment.  The  report  examined  twenty-five  major 
acquisition  programs  whose  development  and  production  costs 
exceeded  $180  billion: 


DOD  has  identified  many  technical  risk  approaches,  both 
quantitative  and  qualitative.  But  there  is  insufficient 
policy  and  training  to  guide  program  managers  in  the 
selection  of  suitable  approaches.  Further,  no  standard 
definition  of  technical  risk  exists  within  DOD. 
Accordingly,  many  program  offices  have  developed  their 
own  informal  definitions  of  technical  risk  and  risk¬ 
rating  categories,  but  GAO  found  them  inconsistent  and 
sometimes  contradictory.  Despite  DOD’s  1981  initiative, 
none  of  the  25  program  offices  had  conducted  a 
quantitative  technical  risk  assessment  to  support 
budgeting  for  risk.  [Ref.  8;p.  3] 


To  date,  the  problems  enumerated  by  the  GAO  in  1986 
remain  largely  unaddressed.  While  a  large  number  of 
logically  consistent  decision  tools  have  been  developed  in 
academic  environments,  accessibility  and  implementation 
by  DoD  acquisition  authorities  and  their  staffs  remain 
inconsistent  or  even  nonexistent. 
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Despite  the  daunting  challenges  involved  in  developing 
such  a  revolutionary  stealth  capability,  the  Navy 
characterized  the  A-12  program  as  possessing  "low"  risk. 

The  Navy  designated  the  T-45  program,  a  jet  training 
aircraft  to  replace  both  the  T-2  Buckeye  and  TA-4  Skyhawk 
training  aircraft,  also  as  "low"  risk  [Ref.  9]. 

The  T-45  was  originally  a  land-based  British  design  that 
had  to  be  altered  for  compatibility  with  the  carrier 
environment.  The  "low-risk"  designation  was  assigned, 
despite  the  fact  that  no  land-based  aircraft  design  had  been 
successfully  reengineered  for  carrier  duty  since  1945.  The 
program  suffered  serious  performance  problems,  schedule 
slippages,  and  cost  overruns  [Ref.  9]. 

The  problem  of  aircraft  acquisition  technical  risk 
assessment  is  not  confined  to  aerodynamic  issues.  The 
Navy's  Enhanced  Modular  Signal  Processor  (EMSP) ,  designed  to 
handle  a  wide  range  of  acoustic  and  electronic  processing 
requirements  through  the  end  of  the  twentieth  century,  has 
had  by  many  problems  despite  its  designation  as  a  "low"  risk 
program  [Ref.  10].  The  Air  Force'  C-17  program  has  been 
beset  by  performance  and  cost  problems  as  well  as  schedule 
delays  arising  from  the  adoption  of  an  ostensibly  "low  risk" 
approach  to  the  software  design  for  flight  control  computers 
[Ref.  11]. 
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Despite  numerous  DoD  and  service-issued  directives  aimed 
at  assessing,  quantifying  and  controlling  risk,  e.g..,  DoD 
Instruction  5000.2,  there  are  no  DoD  mandated  methodologies 
for  meeting  these  directives  [Refs.  12,  13,  14] .  The 
Defense  Systems  Management  College  (DSMC)  has  issued  a 
publication  that  covers  most  of  the  standard  methods  for 
risk  identification  and  assessment  [Ref.  15].  However, 
there  has  been  no  adoption  of  either  the  DSMC's  scale  for 
low,  medium,  or  high  probability  of  occurrence,  nor  any 
policy  guidance  concerning  what  should  constitute  low, 
medium,  or  high  risk  for  acquisition  activities. 

F.  ORGANIZATION  AND  CHAPTER  SUMMARY 

This  chapter  gave  on  overview  of  the  importance  and 
problems  associated  with  the  assessment  of  technical  risk  in 
DoD  acquisition  programs.  The  objective  of  the  thesis  was 
presented,  along  with  limitations  to  problem  definition  and 
subsequent  research.  The  aim  of  the  work  will  be  to  examine 
how  the  tenets  of  Decision  Theory  can  be  applied  to  the 
problem  of  DoD  acquisition  program  technical  risk  assessment 
in  a  manner  that  is  accessible  to  DoD  acquisition  decision¬ 
makers  . 

Chapter  II  will  present  specific  background  information 
and  issues  necessary  for  understanding  the  theoretical  basis 
of  Decision  Theory  and  its  limitations.  Uncertainty  and 
risk  perception  interactions  will  also  be  discussed. 
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Chapter  III  will  l6ok  at  how  technical  risk  assessment 
issues  are  addressed  in  commercial  operations.  Chapter  IV 
will  define  the  DoD  technical  risk  assessment  and  decision 
environment,  a  necessity  for  framing  any  recommendations  for 
potential  risk  assessment  approaches.  Chapter  V  will 
attempt  to  provide  some  recommended  methods  for  technical 
risk  assessment.  Chapter  VI  will  contain  concluding 
remarks,  a  summary  of  major  findings  and  recommendations  for 
future  research. 
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II.  BACKGROUND  AND  THEORETICAL  FRAMEWORK 


A.  THE  DECISION  PROBLEM 

Decision  making  is  a  unique  human  activity.  Only  humans 
seem  to  possess  the  faculty  of  recognizing  the  possibility 
or  availability  of  more  than  one  alternative  or  course  of 
action  in  a  set  of  circumstances  over  some  period  of  time. 
Decision  making  arises  from  the  application  of  some  value 
system  imposed  upon  a  particular  environment.  It  assumes 
that  a  selected  objective  or  goal  can  be  obtained  through 
the  purposeful  expenditure  of  resources  in  hand,  ".  .  .  in 

the  light  of  norms  or  general  principles,  or  of  future 
outcomes."  [Ref.  16: p.  131] 

As  Bunn  points  out,  the  activity  of  decision  making  is 
available  to  individuals,  organizations  and  society. 

Decision  making  is  a  characteristic  of  "purposeful"  systems: 
the  nature  and  gravity  of  decision  making  activities  are 
tied  to  the  attainment  of  objects  or  goals  that  in  some 
fashion  support  or  further  the  "good"  of  the  entity 
formulating  the  decision  [Ref.  17].  Simon's  definition  of 
the  three  phases  of  decision  making  applies  to  the  DoD 
acquisition  environment:  Finding  occasions  for  making 
decisions;  finding  possible  courses  of  action;  and  choosing 
among  courses  of  action."  [Ref.  18]. 
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The  purpose  of  decision  analysis  is  to  provide  a 
logically  consistent  and  flexible  framework  for  helping  the 
decision  maker  generate  and  evaluate  the  various 
alternatives  that  may  present  themselves.  The  flaws 
inherent  in  innate  and  intuitive  decision  making  processes 
have  been  the  subject  of  extensive  research.  Numerous 
researchers  have  commented  upon  and  measured  the  relatively 
limited  human  data  processing  capabilities  (Ref.  19]. 
Decision  types  fall  into  three  categories.  There  are 
automatic  or  intuitive  decisions  that  take  place  in  such  a 
manner  that  the  individual  is  often  unaware  of  the  process. 
Decision  making  in  an  athletic  competition  is  an  example  of 
this  sort  of  decision  making.  Pondering  the  proper  course 
of  action  during  an  athletic  event  almost  invariably  seems 
to  impair  performance. 

Some  decisions  may  be  very  complex  but  exhibit 
characteristics  that  make  their  resolution  tractable  to 
strict  rules  or  instructions.  On  a  mechanical  level,  the 
operation  of  a  thermostat  or  a  fuel  control  unit  falls  into 
this  category.  The  operator  may  not  know  instinctively  what 
action  to  take.  However,  a  satisfactory  result  can  be 
obtained  from  the  careful  and  sequential  following  of  a  set 
of  instructions,  achieving  a  desired  temperature  or  power 
setting  without  weighing  options  or  detailed  process 
knowledge. 
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Lastly,  there  are  those  decisions  that  require  careful 
and  measured  consideration.  These  are  often  characterized 
by  a  bewildering  array  of  possible  alternatives.  Obtaining 
the  knowledge  necessary  to  determine  the  suitability  and 
cost  of  achieving  a  particular  course  of  action  may  be 
extraordinarily  difficult.  Howard  states,  "decision  making 
is  what  you  do  when  you  do  not  know  what  to  do."  [Ref.  20] . 

1.  Elements  of  the  Decision  Problem 

Following  Bunn  [Ref.  17],  four  major  factors 
characterize  the  problem  and  affect  the  subsequent 
formulation  of  any  solution:  Uncertainty,  Multiple 
Objectives,  Multiple  Options,  and  Sequentiality, 
a.  Uncertainty 

Uncertainty  is  the  key  obstacle  to  the 
resolution  of  most  important  decisions  and  affects  the  other 
three  factors.  Many  times  the  decision  maker  can  envision 
multiple  possible  outcomes  and  can  place  some  sort  of  odds 
of  occurrence  on  particular  events.  Some  of  the  outcomes 
will  be  counter  to  the  goals  pursued  and  might  be  ruinous  to 
the  individual  or  organization  making  the  decision. 

Decision  makers  simply  cannot  know,  a  priori,  how  events 
will  proceed  once  a  particular  course  of  action  is  set  into 
mot  ion . 
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b.  Multiple  Objectives 

Organizational  or  individual  goals  may  be 
complex  when  carefully  considered.  The  objective  may  have 
multiple  attributes.  A  computer  system  could  be 
characterized  by  the  length  of  words  it  can  handle, 
processing  time,  or  memory  size.  The  goal  may  be  subject  to 
some  resource  constraint  or  other  conflicting  requirement. 
The  problems  associated  with  nuclear  power  plant  design  and 
siting  [Ref.  21]  are  classic  examples  of  this  dilemma.  How 
should  the  decision  maker  evaluate  independent  and  possibly 
conflicting  goals  and  attributes? 

c.  Multiple  Options 

The  alternatives  that  are  generated  as 
candidates  for  meeting  a  goal  may  not  completely  satisfy  all 
requirements.  Some  combination  of  alternatives  may  be 
required  to  reach  the  objective.  How  should  the  decision 
maker  eliminate  inadequate  options  without  unnecessarily 
reducing  the  resolution  of  any  screening  device?  How  can  he 
ensure  that  he  has  not  overlooked  an  efficient  possibility? 

d.  Sequentiality 

Problems  often  present  themselves  over  a  period 
of  time:  the  entire  issue  is  not  resolvable  at  any 
particular  instant  but  requires  continual  iteration  as  new 
data  become  available.  The  decision  maker  may  be  confronted 
with  numerous  paths  whose  existence  and  efficiency  are 
contingent  upon  previous  actions.  Sequentiality  may  be 
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deliberately  imposed  upon  the  problem  by  the  organization. 

In  his  analysis  of  the  Cuban  Missile  Crisis,  Allison 
[Ref.  22]  discusses  senior  decision  makers*  penchants  for 
delaying  the  implementation  of  decisions  that  may  be 
irreversible  or  delaying  action  because  of  fear  that  the  set 
of  possible  options  may  be  reduced. 

2.  Decision  Analysis  Requirements 

The  purpose  of  decision  analysis  is  to  directly 
address  issues  in  a  manner  that  helps  the  decision  maker 
more  completely  define  his  problem  and  evaluate  his  possible 
courses  of  action.  The  analyst  must  remember  that  someone 
else  will  make  the  decision  and  will  carry  the  onus  of  the 
i^esults  if  events  turn  out  poorly.  But  what  are  the 
requirements  for  a  good  decision  analysis?  First,  the 
analysis  must  capture  the  essential  elements  of  the  problem 
in  a  fashion  that  the  decision  maker  can  grasp.  This 
presupposes  an  interative  format,  relying  upon  the  intuition 
and  judgement  of  those  charged  with  arriving  at  a  decision. 
The  investigation  must  be  couched  in  such  a  manner  that  it 
leads  to  a  complete  understanding  of  the  issue  at  hand,  or 
at  least  as  complete  as  is  possible,  given  the  ever-present 
constraints  of  time  and  money  [Ref.  23]. 

Second,  the  rule  of  coherence  must  be  followed.  If 
the  decision  maker  seeks  to  behave  in  a  rational  manner,  his 
actions  must  be  logically  consistent  and  free  from  hidden 
contradictions.  Rationality  as  a  necessary  condition  in 
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this  sense  implies  that  the  individual  or  other  decision 
making  entity  must  seek  to  maintain  or  improve  his  wealth. 
Failure  to  act  in  a  coherent  fashion  leads  to  the  "perpetual 
money-making  machine"  paradox. 

Suppose  an  individual  considered  event  A  as  less 
likely  than  event  B  and  event  B  as  less  likely  than  event  C. 
Suppose  further  that  rather  than  concluding  that  A  is  less 
likely  than  C,  this  individual  decides  that  C  is  less  likely 
than  A.  Assume  that  event  A  provides  an  outcome  that  is 
valuable  to  this  individual,  a  prize  of  some  sort,  while  B 
and  C  do  not.  In  this  case,  the  person  would  be  willing  to 
pay  some  amount  of  money  to  replace  A  with  B.  If  the  prize 
is  now  contingent  on  B,  the  individual  could  be  induced  to 
pay  some  fee  to  replace  B  with  C.  A  third  sum  of  money 
could  be  obtained  by  offering  to  replace  C  with  A,  an  event 
he  considers  less  likely.  The  individual  is  now  at  the 
starting  point  with  the  exception  that  he  is  demonstrably 
poorer.  As  long  as  the  individual  holds  to  his  order  of 
ranking,  money  can  be  made  ad  infinitum,  or  at  least  until 
he  is  drained  of  cash.  Such  behavior  is  defined  as 
incoherent  [Ref.  17]. 

B.  DECISION  CRITERIA 

Decision  criteria  can  be  separated  into  two  main 
categories,  nonstochastic  and  stochastic.  The  nonstochastic 
approach  ignores  the  existence  of  probability  and  seeks  to 


16 


use  a  restrictive  concept  of  dominance  to  generate  the  best 
solution.  Stochastic  techniques  explicitly  apply  notions  of 
probability  to  the  process  of  determining  a  best  solution. 
There  are  problems  associated  with  the  application  of  both 
these  approaches. 

1.  Nonstochastic  Methods 

Nonstochastic  methods  seek  to  discover  the  best 
alternative  for  satisfying  the  decision  maker's  goals 
through  the  discovery  of  a  course  of  action  that  will  be 
superior  to  all  others,  no  matter  the  outcome  of  events. 
Dominance  approaches  can  be  defined  as  strict  dominance, 
maximin,  maximax,  and  regret  strategies, 
a.  Outcome  Dominance 

Consider  a  matrix  that  lists  possible  results 
for  three  different  alternatives  in  the  event  of  three 
different  outcomes  and  assume  that  the  decision  maker 
prefers  more  to  less: 


POSSIBLE 

OUTCOMES 


ALTERNATIVES 


lA.^ 

A3 

El  12 

1 

3 

E:^  :6 

5 

4 

E3  19 

7 

8 

(1) 
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An  inspection  shows  that  for  any  possible  event, 
El,  Eia,  or  E3,  alternative  A 1  dominates  Az.  As  summarized 
by  Bunn,  any  alternative  Ac,  dominates  Ap>,  if  for  every  Ei 
yic,  is  greater  than  or  equal  to  yip.  and  yic,  is  greater  than 
yip>  for  at  least  one  Ei  [Ref.  17:p.  17].  No  assumptions 
about  the  relative  likelihood  of  event  occurrence  have  been 
made.  While  this  technique  is  simple  to  use,  it  is  only, 
applicable  to  a  very  small  set  of  fortunate  circumstances. 
Additionally,  the  best  this  technique  can  be  expected  to 
provide  is  some  reduction  in  the  number  of  possibly 
satisfactory  alternatives.  As  can  be  seen  from  (1)  above, 
while  alternative  A^i^  can  be  removed  from  consideration, 
deciding  between  and  A3  remains  a  problem, 
b.  Maximin 

The  maximin  approach  takes  a  pessimistic  or 
conservative  approach  to  the  problem  to  the  sorting  of  best 
alternatives.  Consider  (1)  again.  The  maximin  approach  is 
to  search  each  column  and  identify  the  smallest  outcome 
value  a  particular  event  could  generate.  For  this  example, 
the  column  minimums  have  been  identified  by  an  asterisk. 
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ALTERNATIVES 


POSSIBLE 

OUTCOMES 


At  As>  A3 


2* 


3* 


(2) 


The  maximin  solution  problem  is  to  choose  the  alternative 
that  provides  the  largest  of  the  possible  minimum,  in  this 
case  A3.  Maximin  attempts  to  make  the  best  of  a  bad 
situation. 


The  efficacy  of  this  approach  diminishes  rapidly 
in  cases  where  there  are  very  large  differences  in  possible 
payoffs.  Bunn  [Ref.  17;p.  18]  provides  the  following 
counter-example : 


ALTERNATIVES 


1 

1 

As.- 

POSSIBLE 

OUTCOMES 

Ex : 

1 

( 

31 

32 

(3) 

10,000 

33 

Here  the  maximin 

strategy 

would  recommend  As> 

based  upon  a 

strict  observance 

of  the 

decision 

rule.  The 

decision  maker 

following  this  recommendation,  assuming  again  that  the 
object  is  maximize  monetary  returns,  would  end  up  “penny¬ 
wise  and  pound-foolish." 
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Maximax 


c . 

This  strategy  is  the  obverse  of  maximin,  viewing 
the  occurrence  of  random  events  in  the  most  optimistic 
manner.  The  decision  maker  is  required  to  identify  the 
maximum  payoff  for  each  possible  course  of  action  and  then 
select  the  alternative  that  provides  the  greatest  return. 
This  is  clearly  untenable  in  the  event  that  an  alternative 
could  cause  a  very  large  loss.  Consider  the  following 
payoff  matrix: 


POSSIBLE 

OUTCOMES 


ALTERNATIVES 
:  A.  A:^ 

I 

eT:^  31  33 

I 

E=.:  -10,000  33 


(4) 


Maximax  would  recommend  alternative  ,  oblivious  to  the 
potential  for  substantial  monetary  loss. 

d.  Regret 

The  regret  strategy  uses  an  approach 
similar  to  maximin.  The  selection  criterion  focuses  upon 
how  the  decision  might  be  viewed  from  the  vantage  of 
hindsight.  Consider  the  following  two  alternative,  two 
outcome  payoff  matrix; 
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ALTERNATIVES 


I  Ai  As 

I 

POSSIBLE  eTT”?  7 

OUTCOMES  I  (5) 

Es  :  9  8 

A  minimax  regret  analysis  would  consider  what  the  best 
alternative  would  be  for  each  possible  outcome.  Clearly 
if  E.t  occurs,  the  best  strategy  is  select  As.  Likewise,  Ai 
is  best,  if  Es  takes  place.  The  analysis  continues  by 
transforming  the  payoff  matrix  into  a  matrix  of  opportunity 
losses: 

ALTERNATIVES 


:  A  :,. 

1 

1 

A:.. 

POSSIBLE 

E:,'  1  2  ' 

0 

OUTCOMES 

1 

E:2  1  0 

1 

From  this  step,  the  alternative  that  provides  the  minimum 
non-zero  opportunity  loss  is  selected. 

This  approach  has  certain  emotional  appeal, 
particularly  in  situations  where  hindsight  evaluations  can 
severely  affect  the  decision  maker.  It  may  be  the  most 
regularly  applied,  at  least  subconsciously,  of  all  decision 
analysis  tools.  A  serious  flaw  affects  this  strategy.  It 
can  be  shown  that  a  decision  maker  employing  the  minimax 
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regret  approach  could  be  turned  into  a  perpetual  money¬ 
making  machine  [Ref.  24],  violating  the  coherence  principle. 
Additionally,  the  technique  is  subject  to  the  problem  of 
rank  reversal,  which  negates  its  effectiveness  in  tackling 
problems  that  unfold  in  a  sequential  fashion. 

2.  Stochastic  Methods 

The  coherence  principle  implies  that  the  decision 
maker  will  use  all  of  the  information  at  his  disposal.  If 
some  opinion  of  the  relative  likelihood  of  various  events  is 
available,  it  should  be  explicitly  brought  into  the 
analysis.  Information  about  likelihood  could  arise  from  a 
body  of  quantified  statistical  data,  e.g.,  survey  sample  or 
previous  experience,  from  which  a  probability  distribution 
could  be  constructed  or  might  reside  in  the  field  of 
judgement  and  professional  intuition.  When  dealing  with  the 
allocation  of  scarce  resources  to  obtain  some  best  possible 
return,  it  would  be  foolish  not  to  incorporate  as  much 
relevant  information  into  the  decision  making  process  as 
possible.  Three  common  methods.  Modal,  Expected  Value,  and 
Expected  Regret  attack  the  decision  problem  in  a 
probabilistic  manner.  All  are  measures  of  central  tendency, 
and  as  such,  tend  to  ignore  or  reduce  the  impact  of  outliers 
on  the  recommended  decision. 
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a.  Modal  Analysis 


As  the  name  implies,  modal  analysis  seeks  to 
identify  the  most  likely  occurrence,  based  upon  some  set  of 
probabilities  the  decision  maker  has  provided.  Consider  the 
following  example  of  alternatives  and  potential  outcomes 
subject  to  the  assigned  probabilities. 


POSSIBLE 

OUTCOMES 


ALTERNATIVES 


E 

P(EO 

At 

As^ 

A3 

eT 

0.2 

10 

8 

15 

E:^ 

0.5 

1 

6 

9 

3 

E:s 

CO 

o 

18 

5 

1 

Note  that  the  probabilities  sum  to  one.  It  can  be  easily 
seen,  that  under  the  modal  scheme,  Es;  is  most  likely,  hence 
Asr  should  be  the  chosen  course  of  action.  Problems  with 
this  method  arise  when  the  resolution  between  probability 
values  decreases  and  the  potential  payoff  matrix  possesses 
large  differences  between  some  values  and  little  difference 
between  others . 
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Consider  the  counter-example: 

ALTERNATIVES 


POSSIBLE 

E 

P(E.  ) 

A.i. 

Arx. 

A3 

OUTCOMES 

”o72i“~ 

j— 

98 

90 

Era 

0.23 

0 

85 

83 

E3 

0.56 

18 

15 

15 

The  modal  approach  would  yield  E3  as  the  preferred 
alternative,  without  investigating  the  possible  effects  of 
the  large  difference  in  payoff  values.  An  argument  has  been 
made  that  modal  analysis  is  subject  to  the  same  perpetual 
money-making  criticism  that  afflicted  the  minimax  regret 
method  [Ref.  24].  Modal  analysis  can  also  be  affected  by 
the  way  data  are  aggregated.  Consider  the  following  example 
taken  from  Devore  [Ref.  25:p.  11].  The  length  of  service, 
in  years,  for  94  Supreme  Court  Justices  who  had  terminated 
their  services  were  tallied.  If  the  length  of  service  is 
aggregated  by  five  year  increments  the  following  matrix  is 
obtained : 


LENGTH  OF  SERVICE  ! 

NUMBER  OBSERVED 

1 

0-5  1 

11 

6-15  : 

37 

16-25  1 

28 

26  -  35 

17 

>  36  1 

1 

24 


With  this  aggregation,  the  mode  is  6  -  15  years  of  service. 
However  if  the  data  are  processed  in  ten  year  increments  the 
mode  changes. - 


LENGTH  OF  SERVICE 
0-2 
3-7 
8-12 
13  -  17 
18  -  22 
23  -  27 
28  -  32 
>  33 


NUMBER  OBSERVED 

5 
23 
14 
17 
14 

7 

8 

6 


(10) 


The  mode  in  (10)  is  now  3-7  years  of  service.  This 
illustrates  that  the  abilities  of  the  data  to  explain  a 
process  are  sensitive  to  the  metrics  imposed  by  the  decision 
maker. 

b.  Expected  Value 

Expected  value  calculates  the  average  or  mean 
payoff  value  and  then  chooses  the  alternative  that  provides 
the  highest  mean  return.  Each  possible  alternative  payoff 
is  multiplied  by  its  associated  probability  and  the  results 
are  summed  for  each  alternative.  Matrix  (11)  provides  an 
example  of  how  probability  is  incorporated  into  the  payoff 
matrix  to  determine  the  best  alternative. 
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ALTERNATIVES 


POSSIBLE 

OUTCOMES 


E 

P(E.x  ) 

Ai 

As; 

A3 

El 

“  0.2 

“lo" 

8 

15 

E2> 

0.5 

6 

9 

3 

E3 

0.3 

18 

5 

1 

EV(Ai) 

10.4 

7.6 

4 

(11) 


The  expected  value  decision  for  (11)  chooses  option  A.i  as 
the  alternative  that  provides  the  largest  return. 

Since  all  the  events  are  the  result  of  random 
outcomes,  expected  value  analysis  requires  the  decision 
maker  to  be  able  to  withstand  short  term  losses  or  instances 
of  less  than  ideal  returns.  The  expected  value  approach  may 
not  be  appropriate  for  one-of-a-kind  decisions  or  for 
instances  where  exceptionally  large  or  ruinous  losses  are 
associated  with  small,  but  potentially  significant 
probabilities. 

c.  Expected  Regret 

The  expected  regret  method  applies  the 
probabilities  associated  with  each  outcome  to  a  regret  or 
opportunity  loss  matrix  similar  to  (6)  above.  Using  (11) 
and  applying  the  regret  technique,  the  following  payoff 
matrix  is  obtained: 
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ALTERNATIVES 


POSSIBLE 

OUTCOMES 


E  1  P(Ex) 

1 

t 

Ai 

A::^ 

A3 

El!  0.2 

5 

7 

0 

1 

E..!  0.5 

1 

3 

0 

6 

i 

E3:  0.3 

i 

0 

13 

17 

1 

:ER(Ai) 

2.5 

5.3 

8. 

(12) 


Again,  At  is  the  best  alternative  as  it  causes  the  least 
amount  of  opportunity  loss.  Maximizing  EV(Ai)  will  always 
lead  to  the  minimization  of  ER(At).  As  noted  with  expected 
value  calculations,  the  decision  maker  will  be  subject  to 
the  variance  associated  with  each  of  the  probabilities  and 
must  be  able  to  withstand  short-term  instances  of  contrary 
results . 


3 .  Summary 

The  basic  classical  statistical  methods  for 
determining  rational  decisions  were  presented.  Strengths 
and  weaknesses  of  were  noted.  A  limitation  affecting  all 
of  the  methods  was  the  failure  to  account  for  individual 
preferences  in  regard  to  the  magnitude  of  potential  gains 
and  losses.  Methods  that  seeks  to  overcome  this  obstacle, 
while  producing  rational  and  consistent  results  for 
individual  decision  makers,  will  be  presented  in  the 
following  sections. 
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C .  UNCERTAINTY  AND  RISK 

Uncertainty  stems  from  the  problem  of  dealing  with 
random  events  where  the  likelihood  of  an  event's  occurrence 
is  subject  to  some  measure  of  variability.  Humans  have 
exceptional  difficulty  dealing  consistently  with  random 
events.  Coin  tosses  or  other  binomial  events  where  an 
objective  and  precise  probability  of  occurrence  can  be 
specified  are  conceptually  easier  to  handle  than  events  for 
which  no  objective  body  of  data  exists  from  which  to 
generate  a  probability  distribution.  Kahneman  and  Tversky 
[Ref.  26]  have  demonstrated  that  even  individuals  with 
statistical  training  will  often  be  subject  to  three 
particular  biases:  representativeness,  anchoring,  and 
availability.  These  biases  are,  "...  highly  economical  and 
usually  effective,  but  they  lead  to  systematic  and 
predictable  errors."  [Ref.  26: p.  20]  Since  people  have 
difficulty  dealing  with  random  events,  the  concept  of  risk 
is  equally  befuddling. 

The  risk  of  an  event  is  the  probability  of  occurrence 
multiplied  by  cost.  Risk  is  associated  with  notion  of  the 
value  that  may  be  lost  or  unrealized  as  a  result  of  a  random 
event.  An  additional  issue  is  the  size  of  the  variance 
affecting  the  probabilities  applied  to  the  problem.  Because 
values  are  relative,  how  should  a  decision  maker  obtain  a 
defensible  solution  to  the  problem  of  hazarding  current 
resources  against  the  attainment  of  value  at  some  future 


28 


time  when  the  outcome  is  subject  to  randomness?  Extensive 
research  has  shown  that  individual  concepts  of  risk  change 
dramatically  as  the  potential  payoff  or  loss  becomes  very 
large  even  if  the  associated  probabilities  become  relatively 
small.  The  Allais  Paradox,  presented  below,  is  an  example 
of  this  behavior. 

Suppose  a  person  is  offered  the  fol lowing  wager : 

A:  $1  Million  guaranteed 

versus 

B;  $5  Million  with  a  10  out  of  a  100  chance 

$1  Million  with  a  89  out  of  100  chance  (13) 

No  money  with  a  1  out  of  a  100  chance 

Allais,  Raiffa,  and  Tversky  have  shown  that  the  modal 
response  is  wager  B  [Refs.  17,  26,  27].  When  the  situation 
is  altered  so  that  there  are  large  difference  between  payoff 
values  and  small  differences  between  probabilities  people  do 
not  follow  the  expected  value  model.  Compare  the  following 
wager  to  ( 13) ; 

C:  $5  Million  with  a  10  out  of  100  chance 
No  money  with  a  90  out  of  100  chance 

versus  (14) 

D:  $1  Million  with  a  11  out  of  100  chance 
No  money  with  a  89  out  of  100  chance 
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The  expected  value  model  selects  C  over  D  because  of  the 
larger  expected  value: 

EV(C)  »  $500K  >  EV(D)  -  $110K 

The  modal  response  however  is  D.  People  begin  to  focus  on 
the  small  difference  probability  when  the  differences 
between  payoffs  begin  to  get  very  large  [Ref.  283. 

Objective  statements  that  are  applicable  to  a  broad  range  of 
situations  are  exceptionally  difficult  to  make.  The 
following  sections  will  discuss  some  of  the  problems 
associated  with  the  measurement  criteria  that  have  been 
suggested  for  decision  analysis. 

1.  Historical  Antecedents 

Risk  assessment  is  a  very  old  practice.  Records  of 
assessment,  valuation,  and  accounting  procedures,  dating  to 
3200  B.C.  have  been  found  in  the  Tigris-Euphrates  valley 
[Ref.  29] .  The  "risk  accounting"  procedure  that  was  used 
presages  a  modern  proposal  introduced  to  handle  the  risk  and 
uncertainty  issues  surrounding  nuclear  energy  [Ref.  30]. 

The  determination  of  risk  and  subsequent  recommendation  of 
appropriate  courses  of  action  were  placed  in  the  hands  of  a 
priestly  caste  who  used  a  form  of  double-entry  bookkeeping 
to  arrive  at  a  quantified  decision,  carrying  the  weight  of 
assurance  and  confidence.  Modern  antecedents,  employing 
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concepts  of  mathematical  probability  and  scientific  analysis 
of  cause  and  effects  effectively  began  with  Laplace's  1792 
study  of  smallpox  deaths. 

Despite  Laplace's  ground-breaking  effort,  the 
abilities  of  organizations  to  gather  relevant  data  and 
subsequently  manipulate  it  in  mathematically  sound  and 
useful  fashion  remained  haphazard.  Although  the  concept  of 
insurance  to  spread  the  risk  of  commercial  enterprises,  as 
well  as  compensate  for  the  vagaries  of  human  mortality  are 
ancient,  consistent  success  in  risk  assessment  and  survival 
of  risk  managing  entities  required  the  establishment  of 
professional  actuaries.  Covello  and  Mumpower  cite  findings 
of  the  British  Government  in  1867  that  over  75  percent  of 
life  insurance  endeavors  failed ‘in  the  preceding  75  years 
[Ref.  29] .  More  technically  sophisticated  methods  for 
analyzing  the  risk  in  commercial  undertakings  other  than 
life  insurance  or  oceanic  shipping  required  both  more 
rigorous  codification  of  economic  theory  and  more  flexible 
mathematical  tools. 

2.  Modern  Perspectives 

All  of  the  decision  criteria  examined  so  far  either 
ignore  the  existence  of  random  behavior,  or  treat  the 
probabilistic  likelihoods  with  certainty,  failing  to 
consider  the  variable  inherent  in  random  events .  As 
discussed  above,  expected  value  or  expected  regret 
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calculations  fail  to  capture  the  potential  impact  of  the 
loss  of  value  to  the  decision  maker.  The  following  example 
highlights  this  problem. 

Consider  to  wagers  with  identical  probabilities 
of  occurrence: 


Wager  A:  Receive  $10  with  probability  of  0.5 
Lose  $10  with  probability  of  0.5 

(15) 

Wager  B:  Receive  $10,000  with  probability  of  0.5 
Lose  $10,000  with  probability  of  0.5 


Evaluated  from  an  expected  value  perspective,  both  wagers 
are  identical: 


EV(A)  =  EV(B)  =  0. 

However,  if  an  individual  participates  in  wager  B,  the 
magnitude  of  the  loss  may  be  radically  different  from  that 
of  wager  A,  depending  on  what  proportion  of  net  worth  a 
$10,000  loss  represents.  It  is  conceivable  that  an 
individual  could  be  induced  to  pay  sum  amount  of  money  or 
part  with  some  quantity  of  value  to  "buy  out"  of  wager  B  and 
into  wager  A.  How  much  money  an  individual  would  be  willing 
to  give  up  and  under  what  conditions  forms  the  basic 
approach  to  current  risk  analysis  techniques.  A  significant 
obstacle  to  this  search  is  the  variability  of  risk 
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perception  between  decision  makers,  as  well  as,  the 
variability  of  risk  perception  by  individuals  in  different 
decision  environments.  Generically,  endeavors  subject  to 
greater  risk  are  understood  to  possess  either  greater 
likelihood  of  potentially  ruinous  losses  or  instances  where 
the  probability  of  occurrence  may  be  quite  small  but  the 
associated  event  would  be  devastating.  Crossing  a  busy 
highway  blindfolded  would  be  an  example  of  the  former,  while 
an  accident  at  a  nuclear  power  facility  or  an  aircraft 
landing  on  a  house  would  be  examples  of  the  latter  [Ref. 

31].  Risk  analysis  attempts  to  provide  an  additional 
decision  criterion  that  focuses  on  the  potential  variability 
of  an  event,  as  a  complement  to  expected  value  analysis, 
which  ignores  the  variance  issue.  There  are  four  major 
approaches  for  tackling  the  risk  problem: 
a.  Variance 

This  is  the  usual  statistical  definition  of 
estimated  variance: 


•  00 


2 


(y)  dy 


This  captures  the  notion  that  the  greater  the  variance,  the 
greater  the  potential  risk.  It  does  not  address  the  problem 
of  potential  skewness  of  the  distribution  governing  y  as 
Figure  2-1  demonstrates. 
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Figure  2-1  Distributions  with  Same  Means  and  Variances 


If  the  full  probability  density  function  is  considered. 


series  A  would  be  preferred  and  series  B  considered  more 


risky. 

b.  Semi variance 


Semivariance  attempts  to  concentrate  on  the 

perceived  risky  portion  of  the  payoff  distribution, 

using  a  pre-determined  critical  value; 

c 

2 

(y  -  c)  •  f  (y)  <iy 
i 

—CO 
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As  with  the  variance  method,  distributions  with  equal  mean 
and  semivariance ,  but  different  risk  potential,  can  be 
constructed  as  shown  in  Figure  2-2.  As  with  the  example  of 
variance  as  a  measure  of  risk,  when  the  full  probability 
density  functions  of  series  A  and  B  are  considered,  series  B 
would  be  the  riskier  of  the  two. 


SAME  MEAN  AND  SEMIVARIANCE 


i(y) 


-2  -1  0  1  2  S 

y 

""  Series  A  ^  '  Series  B 


Figure  2-2  Distributions  with  Same  Means  and  Semi variance 
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c.  Critical  Probability 


The  critical  probability  approach  uses 
the  Same  format  as  semivariance  but  substitutes  a 
probability  value  for  a  measure  of  variability. 

P(y  <  c)  *  f  (y)  dy 

*  —CD  i 

Bunn  cites  a  definition  generated  by  Fishburn  that  links  the 
two  approaches . 


Ria 

j 

L 


i*C 


I 

i 


b 

(c  -  y)  f 

j 


(y)  dy 


When  b  =  2,  the  semivariance  method  is  used  and  when  b  =  0 
critical  probability  is  operable  [Ref.  17:p.  353.  The  idea 
of  a  critical  probability  has  certain  intuitive  appeal.  The 
critical  probability  can  be  viewed  as  a  goal,  with  reaching 
or  surpassing  the  critical  value  defined  as  success  and 
falling  below  the  critical  value  defined  as  failure.  This 
highlights  the  personal  and  transitory  nature  of  risk 
evaluation  [Ref.  26].  As  with  variance  and  semivariance, 
counter-examples  are  easy  to  create,  as  shown  in  Figure  2-3, 
where  both  distributions  have  equal  area  below  zero  but 
series  B  could  be  considered  more  ris3<y. 
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SAME  MEAN  and  VARIANCE 


Figure  2-3  Distributiona  with  Same  Means  and  Equal  Area 
Below  Zero 

d.  Mean-Variance  Dominance 

This  approach  has  its  roots  in  financial 
analysis.  The  premise  is  the  same  as  discussed  for  the 
nonstochastic  dominance  case.  The  outcomes  of  various 
alternatives  are  characterized  by  their  respective  means  and 
variances.  The  decision  rule  is  to  choose  those  options 
that  possess  the  greatest  mean  and  smallest  variance.  While 
this  may  lead  to  an  "efficient"  set,  where  dominance  cannot 
further  winnow  the  choices,  it  may  not  identify  a  clear 
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winner.  Additionally,  it  requires  a  more  precise  knowledge 
of  the  process  governing  the  outcomes  than  may  be  available 
in  some  decision  environments.  Utility  Theory  offers  a 
potentially  coherent  avenue  around  some  of  these 
limitations. 

D.  UTILITY  THEORY 

In  1944,  John  von  Neumann  and  Oskar  Morgenstern 
published  their  influential  work,  "Theory  of  Games  and 
Economic  Behavior"  [Ref.  5].  Arising  from  a  concept 
originally  put  forward  by  one  of  the  authors  in  1928,  the 
principles  developed  in  their  work  and  embellished  by 
subsequent  researchers  form  the  basis  for  most  current 
techniques  of  risk  assessment.  Morgenstern  and  von  Neumann 
sought  to  develop  a  complete  set  of  rules  that  would  define 
"rational  behavior"  and  could  consequently  be  used  to  guide 
actions  in  any  circumstance.  They  defined  rationality  as 
optimal  behavior  in  an  economic  sense;  individuals  would 
consistently  seek  to  improve  their  conditions.  Optimal 
behavior  could  be  achieved  through  the  use  of  "...  the 
notion  of  mathematical  expectation"  [Ref.  5:p.  32]. 

1.  Axiomatic  Structure 

von  Neumann  and  Morgenstern  define  seven  key 
elements  in  this  theory  which  provide  a  firm,  logical  basis 
of  coherence.  The  theory  is  normative  vice  descriptive, 
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providing  d.  fidmework  that  guarantees  a  rational  result.  In 
its  original  form  it  does  not  necessarily  describe  how  real- 
world  decision  makers  actually  arrive  at  a  solution. 

a.  Structure 

First,  the  decision  maker  can  create  an  order 
out  of  the  decision  situation.  He  can  describe  his 
alternatives  and  can  create  a  listing  of  the  payoffs  that 
would  proceed  from  the  perfect  execution  of  these  options. 
Sophisticated  models  may  be  required  to  capture  the 
essential  elements  of  the  decision  environment,  but  such  a 
construction  is  assumed  to  be  possible. 

b.  Ordering 

Second,  a  preference  order  can  be  determined  for 
the  list  of  possible  objectives  or  outcomes  of  a  specified 
process.  Given  a  choice  of  three  items.  A,  B,  or  C,  with 
the  oppor’tunity  to  select  only  one  at  a  time,  the  decision 
maker  can  order  these  choices  in  a  manner  that  reflects  his 
desires. 

A  >  B  >  C 

This  implies  choice  A  is  preferred  to  choice  B  and  choice  B 
is  preferred  to  choice  C. 
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c.  Transitivity 

The  third  point  is  transitivity.  If  the 
decision  maker  were  provided  with  the  choices  as  pairs,  say 
A  and  B,  or  B  and  C,  subject  to  the  ordering 
A  >  B  and  B  >  C,  then  it  follows  that  A  >  C. 

d.  Indifference 

Fourth  is  indifference.  The  composition  of  the 
choices  may  be  such  that  for  some  particular  grouping,  the 
decision  maker  cannot  categorically  state  a  preference 
between  choices.  This  indifference  or  substitutability 
holds  whether  the  various  pairs  of  choices  are  considered 
alone  or  whether  they  are  grouped  into  a  more  complicated 
option . 

e.  Reduction  of  Compound  Prospects 

Fifth,  "Any  compound  prospect  should  be 

indifferent  to  the  equivalent  simple  prospect  with 
probabilities  computed  according  to  the  usual  rules  of 
probability."  [Ref.  17 :p.  541  This  provides  a  coherent 
structure  to  the  proposed  solutions.  Central  to  this  notion 
is  the  requirement  that  the  decision  maker  has  no  preference 
for  the  time-based  portion  of  an  uncertain  prospect.  This 
rules  out  the  consideration  of  preferences  arising  from  the 
joy  of  playing  the  game,  the  particular  atmosphere 
associated  with  a  particular  decision  environment,  or  the 
manner  in  which  risks  are  presented. 
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f.  Continuity 

Sixth,  utility  values  are  continuous.  This 
involves  the  notion  that  for  extremely  noxious  outcomes,  a 
non-zero  probability  exists  that  would  lead  the  decision 
maker  to  accept  some  other  alternative  which  may  be  trivial 
in  some  other  circumstance.  In  the  normal  course  of  events, 
an  individual  may  be  prefer  a  small  sum  of  money  to  a  zero 
payoff,  say  $5.  Then  a  non-zero  probability  exists,  such 
that,  a  prospect  between  $5  or  death  would  be  attractive. 
While  such  an  extreme  seldom  occurs,  it  is  the  sort  of 
problem  that  explicitly  confronts  public  policy  bodies  when 
ruling  on  issues  that  affect  the  welfare  of  society.  This 
axiom  explicitly  supports  the  construction  of  a  utility 
curve  and  allows  for  the  more  precise  resolution  of 
utilities  in  regions  of  rapidly  changing  risk  perception. 

g.  Monotonicity 

Finally,  preferences  must  be  monotonic.  This 
implies  that  option  A  is  preferred  or  indifferent  to  option 
B,  if  and  only  if,  the  probability  of  A,  Pa,  is  greater  than 
or  equal  to  the  probability  of  B,  p^.  Monotonicity  provides 
a  necessary  and  sufficient  condition  for  the 
preferred/indifferent  ordering  of  various  alternatives. 

Thus,  the  utilities  of  different  alternatives  can  be 
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compared  through  the  use  of  expected  utility.  The  expected 
utility  is  calculated  in  the  same  manner  as  expected  value 
leading  to: 

EU(A)  >  EU(B) 

2.  Limitations 

As  with  any  theory,  there  are  specific  limitations 
that  must  be  observed  when  applying  the  theory 
to  a  particular  decision  problem. 

a.  Utilities  Are  Not  Additive 

In  the  event  that  the  final  payoff  for  some 
prospect  is  the  result  of  some  sequence  of  alternatives,  the 
utility  of  the  final  payoff  is  not  equal  to  the  sum  of  the 
individual  payoff's  utilities: 

U(A  +  B)  7^  U(A)  +  U(B) 

This  is  the  result  of  the  curvature  of  the  utility  function. 
Specifically,  it  recognizes  the  possible  change  in  risk 
perception  governing  alternative  considerations,  as  the 
values  of  the  payoffs  change.  The  individual  utilities  are 
additive,,  only  in  the  case  of  a  risk-neutral  decision  maker 
who  effectively  adopts  an  expected  value  criterion. 
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b.  Strength  of  Preferences  Not  Captured 

The  von  Neumann/Morgenstern  version  of 
utility  theory  creates  an  ordinal  scale  as  opposed  to  an 
interval  scale.  The  numerical  ranking  is  imposed  only  for 
the  purpose  of  sorting  preferences.  Consider  the  example  of 
team  rankings  in  college  football.  A  team  could  be  rated 
fifth  one  year  and  fourth  the  following  year.  This  does  not 
imply  that  the  same  level  of  progress  was  made,  as  a  team 
that  was  ranked  second  one  year  and  then  ranked  number  one 
the  next  year  [Ref.  32], 

c.  Not  Comparable  Between  Decision  Makers 

The  evaluation  of  risk  as  an  integral  part  of 
utility  theory  is  very  personal.  Rapaport  and  Wallenstein 
lidve  shown  that,  .  .  the  concept  of  risk  is  highly 
idiosyncratic,"  and  is  a  function  of  the  risk-taking 
situation  or  environment,  personality  characteristics  of  the 
individual (s )  involved,  and  their  training  and  experience 
[Ref.  333.  This  notion  can  be  extended  to  social  and 
corporate  organizations,  as  well.  A  dramatic  example  of 
this  feature  of  utility  theory  is  MacArthur's  decision  to 
proceed  with  an  amphibious  invasion  at  Inchon,  despite 
persistent  and  strident  staff  recommendations  to  the 
contrary  [Ref.  343. 

Determination  of  the  relevant  probabilities  is  a 
key  to  this  aspect  of  risk  assessment.  As  noted  above, 
decision  makers  can  be  liable  to  the  distortions  caused  by 
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availability,  representativeness,  and  imprecise  notions  of 
causality  [Ref.  26].  Additionally,  decision  makers  often 
discount  assessed  probabilities  once  a  course  of  action  is 
decided  upon,  believing  that  they  possess  the  ability  to 
control  events  [Ref.  35]. 

3.  Certainty  Equivalents 

While  the  axioms  of  Utility  Theory  provide  a 
template  for  determining  whether  or  not  an  uncertain  action 
is  coherent,  some  scheme  is  required  to  help  the  decision 
maker  order  his  preferences  when  faced  with  uncertain 
prospects.  The  use  of  Certainty  Equivalents  was  developed 
for  this  purpo.s'e .  If  the  payoff  from  a  probabilistic 
outcome  can  be  compared  in  some  fashion  with  a  guaranteed 
result,  a  quantifiable  measurement  of  the  individual's  risk 
perception  will  emerge. 

Consider  again  the  wager  proposed  in  the  discussion 
of  modern  approaches  to  risk  assessment: 


Wager  A:  Receive  $10  with  probability  of  0.5 
Lose  $10  with  probability  of  0.5 


(15) 


Wager  B:  Receive  $10,000  with  probability  of  0.5 
Lose  $10,000  with  probability  of  0.5. 


Depending  upon  the  characteristics  of  the  decision  maker,  he 
may  not  be  indifferent  to  these  two  wagers  which  would  be 
classified  as  equal  under  the  expected  value  criterion.  The 
question  then  becomes,  under  what  combination  of 
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probabilities  and  payoffs  would  he  subsequently  be 
indifferent  to  A  and  B?  von  Neumann  and  Morgens tern 
formalized  the  notion  of  certainty  equivalents  as  a  means  of 
addressing  this  issue  [Ref.  5].  The  method  is  intuitively 
easier  to  grasp  if  the  nature  of  the  wagers  is  altered 
somewhat . 

a.  Certainty  Equivalent  Assessment 

Consider  the  following  wager; 

Wager  A:  $1000  with  probability  =  0.5  (16) 

$0  with  probability  =  0.5 


As  the  wager  is  presently  constituted,  the  decision  maker 
might  agree  to  participate  as  there  is  no  potential  loss  and 
it  costs  nothing  to  participate.  Now  suppose  the  wager  is 
amended  as  follows: 


Wager  A;  $1000  with  probability  ^  0.5 
$0  with  probability  =  0.5 


(17) 


Wager  B:  $300  with  probability  =  0.5 
$300  with  probability  =  0.5 


In  this  instance,  the  decision  maker  is  required  to  choose 
between  the  wagers,  one  affected  by  uncertainty,  and  another 
one  a  guaranteed  prospect.  As  presented,  the  decision  maker 
would  choose  alternative  (A) .  Based  upon  an  expected  value 
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calculation  and  dominance  reasoning: 

EV(A)  =  $500  EV(B)  =  $300 

and 

EV(A)  >  EV(B) 

By  similar  dominance  reasoning,  if  the  decision  maker  were 
offered  $600  not  to  participate  in  the  wager,  the  individual 
would  take  the  $600.  Therefore,  there  is  some  value  between 
zero  and  $600  which  if  offered  to  the  decision  maker  as 
certainty,  would  result  in  the  decision  maker  being 
indifferent  to  the  choice  of  either  A  or  B.  Through  an 
iterative  process  it  should  be  possible  to  determine  the 
decision  maker's  certainty  equivalent  for  the  wagers 
presented  in  (17).  For  argument’s  sake,  say  this  value  is 
$400,  i.e.,  if  offered  $400  to  forego  the  wager  between  the 
uncertain  prospect  and  the  guaranteed  payoff,  the  decision 
maker  would  accept  the  $400.  This  idea  can  be  encoded  into 
a  payoff  matrix  as  follows: 


ALTERNATIVES 


POSSIBLE 

E 

P(Ei) 

Ai 

Aa 

OUTCOMES 

eT" 

0.5 

1000 

400 

0.5 

0 

400 

At  a  monetary  certainty  level  of  $400,  the  decision  maker 
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would  be  indifferent  between  A.i.  and  A;^,  Additionally,  the 
sensitivity  of  this  Certainty  Equivalent  (CE)  to  small 
perturbations  could  be  uncovered.  If  the  CE  is  changed  to 
say,  $410,  then  the  decision  might  be  to  choose  Asr- . 
Likewise,  if  the  CE  is  reduced  to  $390,  then  Ai  would  be 
chosen.  A  formal  presentation  of  this  hypothesis  developed 
by  Bunn  is: 

Preference  is  indifferent  between  : 

X  with  probability  p 

or 

Y  with  probability  1  -  p 

and  Z  for  certain  [Refs.  17:p.  41,  5:p.  24], 

The  advantage  of  this  principle  is  that,  in 
theory,  it  provides  a  means  of  uncovering  coherent  choices 
in  the  presence  of  uncertainty.  An  additional  benefit  is 
that  values,  other  than  money,  can  be  used  to  achieve 
coherent  decision  solutions.  Payoff  matrices  more  complex 
than  the  simple  binary  example  presented  in  (18)  can  be 
evaluated  with  this  approach  and  finer  resolution  of  the 
curvature  of  the  utility  function  can  be  obtained  by 
successively  examining  segments  of  an  initially  binary 
problem. 
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b.  Single  Attribute  Utility  Functions 

The  standard  approach  is  to  scale  the  utility 
values  betvv'een  zero  and  one,  with  the  best  case  being 
assigned  a  utility  of  one.  The  resultant  CE  can  then  be 
used  to  further  partition  the  utility  scale  for  as  many 
points  as  desired,  as  demonstrated  by  the  following  utility 
function  constructed  from  wager  (17)  above.  Matrix  (19) 
contains  the  payoff-utility  values  and  Figure  2—4  displays 
the  results  graphically. 


PAYOFF  -  UTILITY  MATRIX 


X  1 

;  u(x) 

1000  ! 

1.00 

700 

1  0.75 

500  1 

;  0.50 

375 

:  0.25 

300  i 

;  0.00 
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Figure  2-4  Single  Attribute  Utility  Function 

As  can  be  see  from  the  graph,  intermediate  utility 
values  can  be  obtained  by  interpolating  between  points.  A 
word  of  caution  is  necessary:  if  the  decision  maker  feels 
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that  great  importance  can  be  attached  to  fitted  utility 
values  in  close  proximity  to  each  other,  then  a  detailed 
sensitivity  analysis  may  be  required  to  obtain  the 
resolution  necessary  to  support  the  decision.  In  such  a 
case,  the  decision  may  not  be  solvable  graphically  [Ref.  17: 
p.  631. 

The  final  benefit  of  a  utility  function  approach 
is  that  the  utilities  can  be  compared  using  Expected  Utility 
values,  EU(x),  thus  capturing  the  risk  assessment  attributes 
of  the  decision-maker.  This  is  accomplished  through  a 
calculation  similar  to  the  determination  of  expected  value. 
The  derived  utility  values  are  multiplied  by  their 
associated  probabilities  and  then  summed  across  the  range  of 
possible  outcomes.  A  coherent  solution  is  that  prospect 
which  has  the  highest  expected  utility  value. 

4.  Risk  Attitudes  and  Premiums 

In  wager  (17)  above,  it  was  posited  that  the  CE 
would  be  $400.  The  expected  value  of  wager  A  was  determined 
to  be  $500.  Can  anything  about  the  decision  maker's 
perception  of  risk  be  determined  from  this  fact?  A  Risk 
Premium  (RP)  can  be  defined  that  is  the  difference  between 
the  CE  assigned  to  a  prospect  and  that  prospect’s  expected 
value: 

RP(x)  =  EV(x)  -  CE(x) 
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If  the  decision  maker's  RP  is  greater  than  zero,  he 
is  considered  risk-averse,  i.e.,  he  would  part  with  some 
value  equal  to  the  RP  in  lieu  of  participating  in  the  wager 
If  the  RP  is  identically  zero,  he  is  risk-neutral.  If  the 
RP  is  less  than  zero,  he  is  risk-seeking,  i . e . ,  he  would  be 
willing  to  pay  some  amount  equal  to  the  RP  in  order  to 
participate  in  the  prospect.  Figure  2-5  provides  a 
graphical  presentation  of  these  three  cases. 


RISK  BEHAVIOR 

UTILITY  vs  CERTAINTY  EQUIVALENT 


U(y) 


Sanaa  A  — Sanaa  B  Sariaa  C 


Figure  2-5  Utility  as  a  Function  of  Certainty  Equivalent 
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Series  A,  the  convex  curve,  is  a  risk-averse 
case.  Series  B,  the  straight  line,  is  the  risk-neutral 
case.  Series  C,  the  concave  curve,  is  the  risk-seeking 
case . 

It  has  been  noted  by  many  researchers  that  risk 
attitudes  are  not  necessarily  constant  over  the  range  of 
possible  payoffs  or  utilities  [Refs.  36,  37,  38].  This  • 
behavior  was  noted  above  in  the  Allais  Paradox  example. 

While  numerous  analytic  descriptions  have  been  formulated  to 
describe  individual  risk-perception  curves  [Ref.  39],  a 
graphical  approach  is  useful  for  depicting  a  decision 
maker's  effective  risk-perception  attitudes.  Once  plotted, 
the  regions  in  the  vicinity  of  inflection  points  can  be  more 
closely  researched  to  provide  feedback  to  the  decision  maker 
about  his  perception  of  the  problem  under  consideration. 

5.  Comments  on  Multi-Attribute  Utility  Theory 

While  the  von  Neumann  -  Morgenstern  formulation  is 
.sufficient  to  handle  individual  attributes  of  a  problem, 
many  decisions  involve  a  combination  of  attributes,  some  of 
them  potentially  conflicting.  The  Multi-Attribute  approach 
developed  by  Raiffa  and  Keeney  extends  the  coherence  of 
Utility  Theory  to  a  broader  range  of  problems.  While 
conceptually  simple  in  formulation,  in  practical  terms  it 
requires  extensive  checks  to  ensure  mutual  preferential  and 
utility  independence  for  all  possible  pair  combinations.  A 
theorem  provided  by  Keeney  and  Raiffa  allows  the  set  of  all 
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possible  pairs  involving  2*"*  -  2  potential  tests  to  be 
reduced  to  n  tests  [Ref.  17].  As  in  the  case  of  single 
attribute  utility  functions,  a  value  for  the  joint  utility 
function  can  be  calculated  for  those  instances  where  the 
joint  utility  function  can  be  decomposed  into  a  linear  or 
multiplicative  form.  The  multiplicative  form  is  applicable 
for  those  cases  where  a  particular  attribute  must  be 
included  in  the  solution  (Ref.  17].  Weighting  factors, 
whose  sum  is  one,  are  used  in  a  manner  analogous  to 
probabilities  in  the  single  attribute  case  to  provide  a 
composite  utility  value  for  some  particular  multi-attribute 
solution  set.  The  various  solutions  can  then  be  compared 
based  upon  a  maximization  rule. 

Other  approaches  to  the  "multi-attribute  problem  have 
been  suggested,  most  notably  Saaty's  Analytical  Hierarchy 
Process  (AHP)  [Ref.  40].  While  easier  to  implement  in 
practice  than  Multi-Attribute  Utility,  these  techniques  are 
subject  to  criticism  on  a  number  of  grounds,  most  notably 
coherence  [Ref.  41] . 

E .  SUMMARY 

This  chapter  has  discussed  the  nature  of  the  decision 
problem  and  the  requirements  that  must  be  met  to  guarantee  a 
rational  or  coherent  solution.  Various  methods  of 
decomposing  the  problem  into  simpler  elements  under 
stochastic  and  nonstochastic  conditions  were  examined.  The 
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difficulties  associated  with  capturing  potential  variability- 
in  uncertain  outcomes  were  presented.  Utility  Theory  was 
presented  as  a  means  of  encoding  an  individual  decision 
maker’s  preferences  and  risk  perceptions  into  a  coherent 
solution.  The  concept  of  Certainty  Equivalents  was 
developed  as  a  method  for  eliciting  preferences  and  risk 
perceptions  consistent  with  the  axioms  of  Utility  Theory. 
Some  limitations  of  the  theory  were  presented  and  the 
general  requirements  for  conducting  analysis  of  multi¬ 
attribute  problems  were  defined.  The  existence  of 
alternative  means  of  analyzing  multi-attribute  problems  was 
noted  along  with  their  potential  problem  of  providing 


coherent  results. 


III.  COMMERCIAL  TECHNICAL  RISK  ASSESSMENT 


A .  BACKGROUND 

The  purpose  of  this  chapter  is  to  examine  some  aspects  of 
technical  risk  assessment  as  practiced  by  commercial 
enterprises.  The  Defense  Science  Board  concluded  in  its 
1985  study  of  Department  of  Defense  development  programs 
that  the  creation  of  the  IBM  360  mainframe  computer,  the 
Boeing  767  aircraft,  the  ATT  telephone  switch,  and  the 
Hughes  commercial  communications  satellite  all  compared,  "in 
complexity  and  size  to  a  major  weapon  system  development, 
yet  each  took  only  about  half  as  long  to  develop  and  cost 
concomitantly  less."  [Ref.  2:p.  49]  There  are  two  questions 
to  consider.  Are  the  risk  assessment  techniques  used  by 
commercial  enterprises  successful,  i.e.,  do  they  provide 
sufficient  information  for  planning  and  system  design? 
Secondly,  can  risk  assessment  techniques  used  in  commercial 
operations  be  applied  to  Department  of  Defense  acquisition 
operations?  The  first  question  will  considered  in. this 
chapter.  The  applicability  of  commercial  techniques  to  the 
DoD  environment  will  be  discussed  in  Chapter  IV, 
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B.  COMMERCIAL  RISK  ASSESSMENT  PRACTICE 

As  discussed  in  Chapter  II,  the  inception  of  modern  risk 
assessment  and  mathematical  probability  theory  began  with 
Laplace's  study  of  smallpox  mortality.  It  was  only  in  the 
mid-nineteenth  century  that  commercial  concerns  developed 
sufficient  understanding  of  the  required  mathematical  tools 
to  begin  analyzing  risk  in  a  quantitative  and  rational 
fashion.  von  Neumann  and  Morgenstern  built  upon  the 
economic  equilibrium  theory  of  the  Lausanne  School  when 
developing  their  Utility  Theory  model  [Ref.  5:p.  15].  Since 
the  publication  of  their  ground-breaking  work  in  1943,  many 
additions  to  inventory  of  risk  assessment  and  decision  tools 
have  been  made.  Farquhar  listed  twenty-eight  different 
techniques  for  tackling  various  aspects  of  the  risk 
assessment  problem  [Ref.  39].  A  natural  expectation  would 
be  that  the  commercial  practice  of  risk  assessment  is  well 
developed  and  that  some  modicum  of  success  is  enjoyed  in 
accurately  identifying  risky  projects.  Upon  a  closer  look, 
it  is  apparent  that  this  is  not  so. 

C.  RISK  AND  AMBIGUITY 

Any  commercial  enterprise  faces  two  key  questions: 

(1)  What  do  consumers  want?  (2)  Can  the  firm  satisfy 
particular  consumer  wants  in  a  profitable  manner? 

Tremendous  resources  are  expended  in  the  attempt  to  define 
consumer  wants  and  preferences  in  everything  from  cars  to 
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political  parties.  However,  as  Kosnik  observed,  ambiguity 
exists  as  to  the  extent  of  consumer  needs.  This  complicates 
corporate  efforts  to  satisfy  these  needs  by  matching 
existing  and  emerging  technologies  with  a  constantly 
changing  marketplace.  The  marketing  approach  views  the 
business  process  as  a  ,  "highly  integrated  effort  to 
discover,  create,  arouse,  and  satisfy  customer  needs  .  .  . 
unfortunately  customers  cannot  articulate  what  they  need." 
[Ref.  42:p.  121]  This  has  led  to  some  inconsistencies  in 
the  application  of  classical  Utility  Theory  as  discussed 
below. 

1.  Ambiguity  Concerns 

Classical  Utility  Theory  is  based  upon  knowledge 
of  the  underlying  probability  distribution  governing  a 
particular  group  of  events.  As  discussed  in  Chapter  II, 
expected  utilities  can  be  calculated  in  these  instances  if 
the  magnitude  of  potential  gains  and  losses  is  known.  A 
rjsk  premium  can  be  specified,  based  upon  the  risk-seeking 
behavior  of  the  firm,  such  that  the  firm  would  be 
indifferent  to  the  outcome  of  the  event  under  consideration. 
However,  if  there  is  ambiguity  affecting  the  probabilities 
governing  the  outcomes,  empirical  evidence  suggests  that  the 
associated  risk  premiums  will  be  larger.  Instances  have 
been  noted  where  the  resultant  risk  premium  for  small,  but 
ambiguous  probability  cases  exceeded  that  applied  to  higher. 
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non-ambiguous ,  loss  probabilities.  In  these  cases,  firms 
seem  to  rely  upon  the  "gut  feelings"  of  the  appropriate 
decision  maker,  than  upon  rigorous  mathematical  analysis 
[Ref.  38] . 

2.  Risk  Perceptions 

The  size  of  the  potential  loss  relative  to  the 
value  of  the  company  is  usually  the  key  feature  in  the 
practice  of  commercial  risk  assessment.  This  is  countered 
by  classical  economic  theory.  Higher  performance  levels, 
i.e.,  greater  profits,  will  accrue  to  firms  that  undertake 
more  risk  [Ref.  36].  The  application  of  probabilities  to 
estimates  of  potential  gains  and  losses  is  the  first  step  in 
analyzing  the  risk  associated  with  a  commercial  undertaking.. 
Those  prospects  possessing  the  largest  expected  value  would 
be  selected  for  exploitation.  However,  in  some  instances, 
firms  may  deliberately  pursue  prospects  having  a  large 
expected  loss  because  of  the  potential  "windfall"  gain  that 
may  accrue.  In  the  long-run  such  actions  would  be  ruinous. 
Tlie  rationale  behind  this  behavior  is  poorly  understood,  but 
these  actions  can  be  seen  by  firms  that  are  in  extremis. 

In  such  an  instance,  the  survival  of  the  enterprise,  as 
perceived  by  management,  may  be  contingent  upon  the  recovery 
of  a  "windfall"  profit  [Ref.  43].  This  seems  to  argue  for 
the  application  of  Utility  Theory  to  commercial  decisions  as 
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opposed  to  the  strict  application  of  Expected  Value.  The 
risk-seeking  behavior  of  the  firm  apparently  changes  with 
its  overall  fiscal  health  [Ref.  38]. 

3.  Problems  with  Utility  Theory 

As  stated  above,  the  Utility  Theory  approach 
implies  that  the  underlying  probability  distribution  can  be 
discovered.  This  can  occur  through  elicitation  of 
management's  subjective  probability  estimates  or  through  the 
careful  investigation  of  some  physical  process  governing  the 
prospects  in  question.  Does  this,  in  practice,  represent 
what  commercial  enterprises  are  doing?  March  and  Shapira 
discovered  that,  while  possessing  risk  preferences, 
commercial  managers  act  in  a  fashion  different  from  the  von  . 
Neumann  and  Morgenstern  model.  ‘Specifically,  managers  were 
willing  to  accept  risks  because  they  did  not  expect  to  bear 
them  [Ref.  38]. 

a.  Perceptions  of  Probability 

Managers  appear  to  be  insensitive  to 
probability  estimates  of  events.  Managers  focus  on  target 
levels  and  make,  "sharp  distinctions  between  taking  risks 
and  gambling."  [Ref.  38 :p.  1404]  As  discussed  in  Chapter 
II,  the  notion  of  a  "reference  gamble"  to  elicit  utility 
values  is  central  to  the  classical  utility  approach. 

Winkler  found  that  if  the  decision  maker  is  an  "expert"  in 
some  field,  then  success  will  be  attributed  to  knowledge. 
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while  failure  will  be  the  fault  of  chance.  Conversely,  if 
the  underlying  process  is  poorly  understood,  then  success 
will  attributed  to  chance  [Ref.  44]. 

b.  The  Value  of  Knowledge 

Given  the  above,  the  assumption  could  be  made 
that  managers  possessing  expertise  in  a  particular  field 
could  sift  through  the  relevant  data  describing  some  process 
and  arrive  at  a  profitable  decision.  This  is  not  always  the 
case.  Capen,  et  al .  conducted  a  study  of  the  profitability 
of  petroleum  companies  when  bidding  for  Government  leases. 
The  problem  studied  by  Capen,  et  al .  was  the  poor  success 
rate  and  subsequently  poor  profitability  of  firms  engaged  in 
Alaska  oil  field  development.  Both  the  Atlantic  Richfield 
Company  and  the  Humble  Oil  Company  had  cooperated  in  the 
investigation  of  the  Alaska  North  Slope  oil  fields.  The 
relevant  information  about  parcel  petroleum  characteristics 
was  pooled  and  known  to  both  companies.  However,  during  the 
bidding  process  for  subsequent  field  development,  extremely 
large  bid  variances  for  the  same  parcels  existed.  The 
authors  cited  ratios  of  100  for  some  parcels,  with  5-10 
being  the  most  comraon.  The  authors  concluded  that  the 
winner  of  a  particular  parcel  was  the  firm  that,  ’’most  over¬ 
estimates  the  true  tract  value."  [Ref.  45 :p.  643] 

Access  to  the  same  body  of  data  and  analysis  by  highly 
skilled  professionals  did  not  guarantee  success.  The  value 
experts  placed  upon  a  particular  tract  was  significantly 
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different.  Thus,  recourse  to  expert  knowledge  in  estimating 
the  probability  of  success  in  a  venture  does  not  guarantee 
the  expected  pay-off. 

D.  TECHNICAL  RISK  ASSESSMENT  ISSUES 

The  above  quotation  from  the  Defense  Science  Board  (DSB) 
implied  that  commercial  firms  may  generally  be  successful  in 
their  analysis  of  the  technical  risks  affecting  the 
implementing  of  new  ideas.  While  the  projects  cited  by  the 
DSB  were  resounding  successes  for  their  manufacturers, 
successes  of  this  magnitude  are  not  commonplace. 

1 .  Accuracy  of  Assessment 

The  RAND  Corporation  conducted  a  study  of 
chemical  pioneer  process  plants.  The  corporations  building 
the  pioneer  plants  were  all  successful  operations, 
possessing  large  capital  and  information  resources,  and 
management  structures  attuned  to  the  strategic  challenges 
facing  their  companies.  The  study  concluded  that 
satisfactory  performance  required  in  excess  of  ten  years 
additional  development  work  and  that  large  cost  overruns 
where  common.  The  study  attributed  this  to  the 
establishment  of  fixed  budget  values  prior  to  the  completion 
of  a  final  design  and  incomplete  understanding  of  the 
challenges  to  be  surmounted  [Ref.  45]. 
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Similar  findings  were  reported  by  Davis.  Initial 
construction  cost  estimates  for  new  facilities  were 
typically  less  than  half  the  eventual  amount.  Davis 
attributed  this  to  under-estimating  costs  as  opposed  to  true 
cost  overruns.  Additionally,  eighty  percent  of  new  projects 
failed  to  achieve  their  predicted  market  share  [Ref.  47].  A 
study  by  Battel le  Memorial  Laboratories  discovered  an 
average  of  19.2  years  between  invention  and  commercial 
production  [Ref.  48]. 

2.  Technological  Uncertainty 

A  slightly  different  conclusion  was  reached  in  a 
recent  survey  of  108  San  Francisco  based  companies  spanning 
service -based  to  high  technology  firms.  The  ability  to 
accurately  ascertain  the  difficulties  associated  with  new 
ventures  were  more  acute  for  organizations  relying  upon 
advanced  technologies,  irrespective  of  company  size.  The 
more  advanced  the  technology  providing  the  company's 
profits,  the  greater  the  potentially  negative  consequences 
of  inadequate  technical  risk  assessment.  The  higher  the 
level  of  technological  advancement,  the  higher  the  level  of 
opportunity  costs  for  a  successful  decision.  Greater 
technological  coiriplexity  implies  a  greater  degree  of 
uncertainty  or  ambiguity  for  the  key  decision  maker 
[Ref.  49] . 
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Ghemawat  agrees  with  this  conclusion.  High 
technology  companies  are  faced  with  the  dilemma  of  choosing 
between  a  high-risk,  high-return  prospect  or,  "foregoing  a 
head  start  (and  also  experience  related  cost  advantages)  by 
waiting  until  technological  uncertainty  is  resolved." 

[Ref.  50:p.  148] 

3 .  Summary 

Commercial  operations  have  had  minimal 
success  in  solving  the  probability  estimation  problems 
cataloged  in  Chapter  II.  Corporations  are  hampered  by 
judgement  biases  which  result  in  schedule  delays  and  cost 
overruns.  Issues  of  risk  assessment  and  technological 
uncertainty  are  as  severe  for  firms  marketing  high 
technology  products  as  for  DoD  acquisition  decision  makers 
attempting  to  maintain  a  technologically  superior  force. 

Risk  assessment  methods  that  have  proven  to  be  successful 
are  not  suitable  for  firms  operating  in  very  dynamic 
markets. 

E.  EVALUATION  TECHNIQUES 
1.  Discounted  Cash  Flows 

The  primary  method  for  evaluating  the  potential  of 
any  project  is  through  the  use  of  Discounted  Cash  Flows 
(DCF).  The  basic  premise  is,  "a  dollar  today  is  worth  more 
than  a  dollar  tomorrow."  [Ref.  52]  In  theory,  the  DCF 
approach  provides  a  rational  and  sound  method  for  estimating 
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the  cash  generated  by  a  project  during  different  time 
periods.  Projects  competing  for  limited  corporate  dollars 
can  be  readily  compared  with  this  technique.  Additionally, 
a  firm  can  compare  the  projected  cash  flow  of  a  project 
against  what  the  firm  could  earn  by  simply  putting  the  money 
in  the  bank  or  through  the  use  of  similar  financial 
instruments.  The  DCF  method  is  not  foolproof.  It  relies 
upon  accurate  and  unbiased  estimation  of  three  key  factors: 
Inflation,  riskiness  of  the  project  over  its  projected  life, 
and  risk  reduction  through  diversification. 

a.  Inflation 

Changes  in  the  monetary  inflation  rate  can 
dramatically  affect  the  profitability  of  a  project.  The 
effects  of  inflation  become  more  pronounced  as  the  time 
liori'zon  of  the  projects  expected  life  becomes  longer.  While 
an  accurate  estimate  of  inflation  rates  is  crucial,  the  firm 
is  limited  in  its  ability  to  control  the  effects  on  cash 
flows.  Uncertainty  in  this  regard  tends  to  push  firms 
toward  projects  having  quick  payoffs  [Ref.  51]. 

b.  Project  Risk 

The  technical  risks  associated  with  any 
project  decrease  over  time.  This  is  a  natural  result  of  the 
learning  process.  Maintenance  of  a  fixed  risk  factor  for 
the  life  of  the  project  tends  to  under-value  the  cash  flows 
that  would  occur  as  the  project  reached  maturity.  Older 
technology  does  not  necessarily  imply  useless  technology. 
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Large-scale  projects  can  continue  to  provide  significant 
cash  flows  long  after  the  original  payback  has  been 
satisfied  [Refs.  51.  52].  A  further  complication  arises 
from  the  methods  used  by  firms  to  estimate  the  probabilities 
of  high  risk  events.  Firms  operating  in  established,  slowly 
growing  markets  have  enjoyed  success  by  employing  these 
techniques.  However,  "for  fast  moving  and  world-wide 
industries,  they  have  been  a  disaster."  [Ref.  53:p.  124] 
c.  Diversification 

A  key  tool  available  to  commercial  operations 
is.  diversification.  While  there  are  many  diversification 
techniques,  the  basic  concept  is  that  of  portfolio 
management.  The  comrriodity  future  markets  are,  in  principle, 
an  example  of  this.  An  individual  takes  actions  such  that 
potential  losses  are  balanced  by  gains,  so  that  the  basic 
financial  position  is  protected.  Futures  contracts  are 
bought  or  sold  to  match,  hedge,  the  individual ' s  current 
holdings  and  risk  perceptions.  The  purchase  of  life  and 
property  insurance  policies  are  other  examples. 

Corporations  can  manage  their  various  risk  exposures  through 
the  use  of  financial  instruments  or  through  investing 
internally  in  a  range  of  projects  possessing  varying  degrees 
of  risk.  The  idea  is  to  compensate  for  risk  in  a  way  that 
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maintains  the  company's  desired  risk-seeking  position. 
Companies  desiring  a  low  risk  position  would  not  invest  in 
programs  having  potentially  high  risks  [Refs.  51,  36]. 

2.  Simulation  Techniques 

A  modification  of  the  basic  DCF  approach  is  to  apply 
a  Monte  Carlo  simulation  to  the  firm's  estimation  of  its 
business  environment.  Many  software  programs  are  available 
that  allow  the  user  to  specify  the  nature  of  the  underlying 
probability  density  function.  A  truncated  log  normal 
distribution  is  typically  applied  to  new  projects.  Its 
shape  matches  the  expected  occurrence  of  budget  overruns 
versus  underruns:  Overruns  at  multiples  of  the  original 
estimate  are  far  more  likely  than  very  large  underruns  [Ref. 
52].  Wliile  this  approach  allows  a  firm  to  quickly  examine  a 
multitude  of  scenarios  with  varying  risk  estimates,  the 
basic  problem  of  determining  the  relevance  of  the 
probabilities  applied  to  the  problem  remains.  The  numbers 
produced  are  subject  to  the  biases  and  misunderstandings 
examined  in  Chapter  II.  Strassman  admits  that  the 
probabilities  of  occurrence  that  he  uses  are  based  upon 
personal  experience  and  are  not  empirically  defensible  [Ref. 
52].  The  simulation  approach  simplifies  the  "number- 
crunching"  aspect  of  analyzing  a  lot  of  prospects,  but  does 
not  necessarily  deliver  an  unbiased  result. 
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a.  Risk  Adjusted  Discounted  Cash  Flow 

Strassman  advocates  the  use  of  Risk  Adjusted 
Discounted  Cash  Flow  (RADCF)  analysis  [Ref.  52].  This 
approach  splits  the  DCF  into  two  segments:  financial 
and  operational  risk.  Operational  risk  covers  all  risks 
other  than  those  associated  with  the  cost  of  capital 
[Ref.  52].  Low  risk  projects  are  those  possessing  at  least 
a  97.5  percent  chance  of  providing  profits  over  the  life  of 
the  project.  High  risk  projects  are  those  with  less  than  or 
equal  to  a  2.5  percent  chance  of  providing  profits  over  the 
life  of  the  project.  Moderate  risks  fall  between  these  two 
extremes  [Ref.  54:p.  19].  While  providing  better 
quantitative  data  for  decision  makers,  this  approach  still 
begs  the  question  of  how  applicable  probabilities  should  be 
determined. 

3.  The  Venture  Capitalist  Approach 

The  Venture  Capitalist  represent.s  a  different 
entity  than  the  standard  commercial  firm.  While  interested 
in  profits,  he  does  not  necessarily  have  to  worry  about 
defending  market  share  or  finding  new  uses  for  his  inventory 
of  technologies.  The  Venture  Capitalist  is  generally  free 
to  pick  his  area  of  exploitation  and  arranges  the  timing  and 
scope  of  his  efforts  accordingly. 
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a.  Decision  Criteria 


A  Venture  Capitalist  typically  looks  at  three 
items:  business  potential,  i.e.,  market,  pending  solution  of 
technical  problems,  expertise  and  commitment  of  the 
development  team,  and  lastly,  financial  requirements  [Refs. 
55,  56]  . 

b.  Number  of  Projects 

A  typical  Venture  Capitalist  will  look  at  two 
to  three  hundred  prospects  a  year.  Perhaps  ten  percent  of 
these  will  be  selected  for  further  consideration.  Actual 
project  commitments  will  be  only  two  or  three  [Refs.  55, 

561.  The  average  Venture  Capitalist  will  be  involved  in  a 
total  of  five  to  seven  projects  at  any  time.  The  size  of 
the  commitment  is  dependent  upon  the  Venture  Capitalist's 
familiarity  with  the  technologies  in  question  and  his 
estimate  of  the  technical  expertise  of  the  development  team. 
Financial  arrangements  often  call  for  those  whom  the  Venture 
Capitalist  is  backing,  to  invest  a  substantial  portion  of 
their  net  worth  in  the  project  [Refs.  55,  56]. 

c.  Expectations 

The  goal  of  the  Venture  Capitalist  is  to 
quickly  turn  the  project  into  a  successful  business  venture 
and  then  divest  himself  of  the  asset.  Despite  the  careful 
selection  of  projects  and  the  relatively  large  commitment  of 
funds  and  personal  effort,  on  average,  the  successful 
Venture  Capitalist  can  expect  about  twenty  percent  of  his 


68 


projects  to  be  complete  failures  and  about  twenty  percent  to 
meet  his  profit  goals.  The  remaining  sixty  percent  will 
fall  in  the  middle,  requiring  additional  time  to  reach 
business  goals  and  substantial  divestiture  effort  [Refs.  55, 
56] .  These  results  seem  to  match  those  of  the  larger,  more 
conventional  firms  discussed  above  [Refs.  46,  48,  53]. 

F.  SUMMARY 

This  chapter  examined  the  ways  in  which  commercial 
operations  seek  to  assess  their  risk  exposure.  The  use  of 
classical  statistical  tools  and  Utility  Theory  was 
discussed.  Some  of  the  common  evaluation  techniques  were 
described,  and  their  strengths  and  weaknesses  noted.  The 
susceptibility  of  commercial  operations  to  biases  in 
judgement,  and  over-estimation  of  the  occurrence  of 
favorable  events  described  in  Chapter  II,  were  highlighted. 
While  quantitative  approaches  to  technical  risk  are 
available,  methods  for  probability  estimation  rely 
exclusively  upon  individual  judgement  and  assessment. 
Standard  probability  estimation  methods,  e.g.,  reference 
gambles  or  probability  wheels,  are  spurned  by  corporate 
decision  makers.  Risk  compensation  and  diversification  was 
also  discussed. 
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IV.  DoD  ACQUISITION  DECISION  ENVIRONMENT 


A.  INTRODUCTION 

The  Department  of  Defense  acquisition  process  is 
exceedingly  complex.  Managers  and  decision-makers  struggle 
to  achieve  the  technical  performance  characteristics 
required  to  support  combatant  forces  within  schedule  and 
cost  constraints.  They  must  deal  with  a  host  of  agencies 
and  oversight  bodies  charged  with  monitoring  all  aspects  of 
cost,  schedule,  and  performance  progress.  Acquisition 
decision-makers  must  be  responsive  to  the  directives  and 
requests  for  information  from  various  service.  Department  of 
Defense,  and  Congressional  entities  that  may  have  an 
interest  in  some  portion  of  the  acquisition  process.  In 
1985,  the  Pentagon  submitted  almost  24,000  pages  of 
documentation  to  Congress,  "stemming  from  prior  years’ 
defense  authorization  and  appropriations  bills  and  their 
accompanying  reports."  [Ref.  57: p.  76]  Between  1970  and 
1985,  the  number  of  congressional ly  mandated  reports 
increased  1000  percent  [Ref.  57:p.  76]. 

The  Department  of  Defense  acquisition  process 
encompasses  a  multitude  of  functional  disciplines  other  than 
the  contracting  for  a  particular  piece  of  hardware  or 
software.  Careful  consideration  must  be  given  to  areas  such 
as  maintenance  and  logistics  support  over  the  life  of  the 
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system.  The  number  of  personnel  qualifications  and  training 
requirements  for  operators  and  maintenance  personnel  must  be 
taken  into  account.  Basing  requirements,  support 
facilities,  and  environmental  impact  of  the  new  system  or 
its  production  process  have  to  be  planned.  Unique  test  and 
evaluation  facilities  may  be  required  to  ensure  that  an 
accurate  estimate  of  the  system's  real  operational 
performance  is  obtained.  The  interoperability  of  the 
equipment  with  other  services  or  U.S.  allies  must  be  weighed 
during  the  definition,  design,  and  testing  of  new 
capabilities. 

An  in-depth  analysis  of  all  aspects  of  the  acquisition 

process  is  outside  the  scope  of  this  thesis.  This  chapter 

will  focus  only  on  the  assessment  of  technical  risk  and  its 

impact  on  the  overall  acquisition  process  as  described  in 

DoD  Instruction  5000.2,  "Defense  Acquisition  Management 

Policies  and  Procedures".  [Ref.  12]  This  docciment  requires 

that  essential  program  elements 

"Include  clearly  defined  criteria  for  elements  leading 
to  the  risk  assessment  events.  The  satisfaction  of 
these  criteria  must  be  documented  to  support  the  rigor 
necessary  in  the  risk  assessment  process." 

[Ref.  12:p.  5-B-2] 

The  intent  here  is  to  provide  a  broad  overview  of  the 
decision  criteria  mandated  by  the  "5000"  series  documents 
[Refs.  12,  13,  14]  and  highlight  the  essential  chronological 
features  facing  the  acquisition  decision-maker. 
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B.  ACQUISITION  PROCESS  OVERVIEW 

Office  of  Management  and  Budget  Circular  A-109 
[Ref.  58],  specifies  requirements  necessary  for  the  conduct 
of  any  federal  acquisition.  DoD  Directive  5000.1  and  DoD 
Instruction  5000.2  are  the  governing  documents  for  DoD 
acquisition  decision-makers.  They  detail  the  procedures 
that  must  be  followed  and  the  objectives  that  must  be  met 
for  a  major  new  system  start  or  an  upgrade  to  a  system's 
existing  capabilities. 

1.  Decision  Makers 

The  Under  Secretary  of  Defense  for  Acquisition  is 
the  Defense  Acquisition  Executive  (DAE) .  He  is  supported  by 
the  Defense  Acquisition  Board  (DAB) .  The  DAB  is  the  top- 
level  review  body  for  major  weapons  systems  acquisition 
decisions.  The  DAB  is  chaired  by  the  DAE.  Other  members  of 
the  Board  include  the  Deputy  Under  Secretary  of  Defense  for 
Acquisition;  Service  Acquisition  Executives  of  the  Military 
t^epartments ;  Director  of  Defense  Research  and  Engineering; 
the  Assistant  Secretary  of  Defense  for  Program  Analysis  and 
Evaluation;  the  Comptroller  of  the  Department  of  Defense; 
and  the  Director  of  Operational  Test  and  Evaluation.  The 
Vice  Chairman  of  the  Joint  Chiefs  of  Staff  serves  as  Board 
Vice  Chairman  [Ref.  12:p.  13-A-23 . 
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a.  Supporting  Committees 

The  DAB  is  supported  by  three  committees: 
Strategic  Systems  (SSC) ;  Conventional  Systems  (CSC);  and  C3I 
Systems  (C3IC) .  The  committees  provide  specific  expertise 
to  support  the  acquisition  review  process  (Ref.  59:p.  3]. 

b.  DAB  Responsibilities 

ITie  DAB  convenes  for  each  milestone  review  to 
ensure  compliance  with  previously  specified  performance 
objectives.  Additionally,  the  DAB  is  to  act  as  an 
independent  assessor  of  program  health  and  future  viability. 
The  DAB  provides  recommendations  to  the  Under  Secretary  of 
Defense  for  Acquisition  on  the  various  performance,  cost, 
and  schedule  trade-offs  that  may  be  proposed  by  the  new 
system's  program  manager.  Consideration  of  relevant  risk 
areas,  threat,  technology,  design  and  engineering,  etc.,  is 
an  integral  portion  of  this  process  during  Milestone  review 
[Ref.  12 :p.  13-A-2,  Ref.  13 ;p.  4-E-l ] .  Specific  information 
on  the  mechanics  of  the  decision  process  are  provided  in  the 
following  section. 

2.  Decision  Process 

DoD  uses  a  phased,  systematic  approach  to  analyze 
the  military  requirements  problem,  develop  satisfactory 
solutions  that  meet  performance  goals,  and  manufacture  and 
support  an  operational  system  in  a  timely  fashion  at  an 
affordable  cost.  Five  basic  milestones  are  imposed  on  the 
acquisition  process  to  ensure  that  these  requirements  are 
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met.  Milestones  are  paired  with  follow-on  phases.  The 
phases  are  structured  toward  developing  an  operational 
system  while  satisfying  the  interim  goals  set  hy  the  DAE  and 
the  DAB.  The  goal  is  to  develop  an  event-driven  acquisition 
strategy,  "that  links  program  decisions  to  demonstrated 
accomplishments  in  development,  testing,  and  production." 
[Ref.  59 :p.  2]  This  structure  is  discussed  below, 
a.  Mission  Need  Statement 

The  acquisition  process  begins  with  the 
determination  of  the  basic  system  performance  requirements 
necessaiy  to  either  support  new  military  obligations,  combat 
emerging  threat  capabilities,  or  take  advantage  of  new 
technologies  that  could  substantially  reduce,  "ownership 
costs  or  improve  the  effectiveness  of  existing  materiel . " 
[Ref.  12 :p.  3-2]  The  procurement  process  for  new  capability 
only  begins  after  a  determination  that  military  requirements 
cannot  be  met  via  changes  to,  "doctrine,  operational 
concepts,  tactics,  training,  or  organization."  [Ref.  12: 
p.  3-2]  These  requirements  are  formulated  in  a  "Mission 
Need  Statement"  (MNS) . 

(1)  Mission  Need  Validation.  Mission  needs 
identified  at  lower  echelons  are  validated  by  a  designated 
"operational"  authority:  a  Unified  Command,  e.g.,  Atlantic 
Command,  a  Specified  Command,  e.g..  Aerospace  Defense 
Command,  the  Military  Departments,  the  Office  of  the 
Secretary  of  Defense,  or  the  Joint  Staff  [Ref.  12:p.  3-2]. 
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If  the  validated  requirement  could  potentially  involve  the 
use  of  new  technologies  or  require  a  major  upgrade  to 
existing  performance  capabilities,  then  the  requirements 
also  must  be  validated  by  the  Joint  Requirements  Oversight 
Council  (JROC) . 

(2)  Performance  Requirements.  The  intent  of  the 
MNS  is  to  capture  perceived  operational  requirements  in  a 
broad  fashion.  The  statement  should  be  sufficiently 
detailed  to  enable  subsequent  technical  and  engineering 
studies,  but  must  avoid  identifying  a  specific  system 
solution  to  the  military  requirements  [Ref.  13:p.  2-1-1]. 

As  an  example,  a  Mission  Need  could  be  generated  for  the 
collection,  evaluation,  and  distribution  of  high  quality 
battlefield  imagery,  without  specifying  the  use  of  a 
particular  technique.  The  MNS  would  specify  the  resolution 
required,  size  of  imagery  field  of  view  provided  to 
combatant  forces,  data  transmission  rate,  cryptographic 
requirements,  etc.  It  would  then  become  the  responsibility 
of  the  relevant  DoD  acquisition  element  to  investigate  and 
develop  feasible  technical  approaches  that  meet  the  military 
requirements  consistent  with  schedule  and  cost  constraints. 
DoD  will  often  engage  industry  to  supplement  or  fulfill 
these  requirements. 

(3)  Threat  Assessment.  DoD  Manual  5000. 2-M 
specifically  requires  that  the  Mission  Need  Statement 
identify  the  specific  threat  to  be  countered  and  provide  a 
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description  of  the  likely  threat  environment  [Ref.  13: 
p.  2-1-1].  This  information  is  used  at  later  milestones  to 
validate  the  sufficiency  of  proposed  technical  approaches 
and  help  ensure  consistency  of  evaluations. 

(4)  Joint  Requirements  Oversight  Council.  The 
purpose  of  the  MNS  review  by  the  Joint  Requirements 
Oversight  Council  (JROC) .  is  to  ensure  development  of 
capabilities  that  may  benefit  more  than  one  service  or 
enhance  overall  force  qualities  in  a  rational  manner,  while 
accounting  for  the  requirements  of  joint  operations  and 
deployment.  Validation  of  the  requirements  set  forth  in  the 
MNS  is  also  an  integral  part  of  this  review. 

b.  Milestone  0:  Concept  Studies  Approval 

Upon  validation  of  a  legitimate  military  need, 
the  Defense  Acquisition  Executive  (DAE)  will  direct  the 
initiation  of  concepts  studies.  The  purpose  is  to  begin  the 
investigation  and  research  process  necessary  to  define  the 
specific  Lechnical  characteristics  or  ''concepts”  a  new 
system  must  possess  to  satisfy  the  MNS.  The  DAE  will  issue 
an  Acquisition  Decision  Memorandum  (ADM) ,  specifying  the 
minimum  number  of  possible  ideas  that  should  be  investigated 
and  identifying  the  lead  DoD  agency  responsible  for 
conducting  the  necessary  research  [Ref.  59;p.  5].  At  this 
juncture,  no  definite  date  is  established  for  the  discovery 
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of  a  set  of  satisfactory  solutions  but  a  minimum  set  of 
requirements  will  be  established  for  progression  to  the  next 
milestone. 

c.  Phase  0:  Concept  Exploration  and  Definition 

Successful  completion  of  Milestone  0  marks  the 
formal  beginning  of  the  "Concept  Exploration  and  Definition" 
phase.  The  purpose  of  this  phase  is  to  identify  a  set  of 
possible  technical  solutions  to  the  MNS  requirements.  The 
accuracy  of  the  threat  environment  posed  by  the  original  MNS 
and  the  requirements  for  a  new  capability  are  also 
investigated.  Additional  Phase  0  objectives  are  to  meet 
feasibility  requirements  specified  by  the  DAE  so  that  the 
program  may  progress  to  Milestone  I  and  to  begin 
the  development  of  an  acquisition  strategy  [Ref  12: 
p.  3-8]  . 

d.  Milestone  I:  Concept  Demonstration  Approval 

Successful  completion  of  Milestone  I  marks  the 
official  beginning  of  a  new  DoD  acquisition  program.  The 
purpose  of  this  checkpoint  is  to  ensure  that  the 
requirements  specified  at  Milestone  0  have  been  met.  The 
most  promising  candidate  solutions  to  the  MNS  are  considered 
along  with  their  availability.  The  accuracy  of  the 
projected  threat  environment  is  validated  and  the  military 
requirements  contained  in  the  MNS  examined.  A  complete  set 
of  acquisition  documentation  must  be  submitted  at  this 
review,  including  the  Integrated  Program  Summary  (IPS).  The 
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IPS  contains  seven  annexes  addressing  all  relevant  portions 
of  the  acquisition  process.  At  this  juncture,  technical 
risk  is  specifically  considered  in  Annex  D  to  the  IPS 
[Ref  13:p.  4-E--1]. 

As  mentioned  in  the  previous  section,  the 
program  manager  is  required  to  categorize  the  risks  of 
threat,  technology,  design  and  engineering,  etc.  A  ranking 
scale  of  low,  medium,  or  high  is  used.  Numerical  data  is 
not  required.  No  guidance  is  provided  as  to  what 
constitutes  admission  into  one  of  the  categories.  The 
technical  approach  chosen  will  determine  the  magnitude  of 
potential  gains  or  losses  and  the  likelihood  of  their 
occurrence. 

The  ADM  issued  at  completion  of.  Milestone  I  will 
specify  a  concept  baseline  containing  initial  cost, 
schedule,  and  performance  objectives.  Exit  criteria  will  be 
specified.  These  will  form  an  interim  set  of  goals  to  be 
achieved  during  Phase  I. 

e.  Phase  I;  Demonstration  and  Validation 

The  purpose  of  Phase  I  is  to  improve  the  design 
characteristics  and  better  define  the  expected  capabilities 
of  the  system.  Efforts  are  made  to  improve  the  design 
team's  understanding  of  the  technical  processes  involved  in 
the  selected  approach (s) .  Requirements  necessary  for  the 
successful  completion  of  the  Milestone  II  review  are 
pursued.  A  Development  baseline  for  the  most  promising 
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alternative  is  created  during  this  phase.  High  risk  areas 
and  their  potential  compensators  are  to  be  identified  as 
well  [Ref.  12:p.  3-14] 

f.  Milestone  II:  Development  Approval 

The  key  aspect  of  this  milestone  is  to  determine 
whether  the  results  of  the  Demonstration  and  Validation 
phase  support  continuation  of  the  program.  Specific  areas 
of  concern  at  this  juncture  are  the  validity  of  the 
potential  threat  assessment  and  countering  Mission  Need  and 
whether  or  not  the  proposed  technical  solutions  are 
understood  and  actually  employable.  Technical  risk  problems 
can  significantly  delay  passage  of  this  milestone,  resulting 
in  potential  trade-offs  between  original  performance,  cost, 
and/or  schedule  goals.  An  update  IPS  with  revised  risk 
assessment  information.  Annex  D,  is  provided.  Resource 
allocation  in  terms  of  personnel  and  money  are  reviewed. 
Successful  passage  of  this  checkpoint  results  in  a  further 
refinement  of  base-line  cost,  schedule,  and  performance 
goals.  The  ADM  will  address  specific  criteria  that  must  be 
met  for  passage  of  Milestone  III,  Production  Approval  and 
this  ADM  approves  entry  into  Phase  II:  Engineering  and 
Manufacturing  Development. 

[Ref.  12:p.  3-18] 
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g.  Phase  II:  Engineering  and  Manufacturing 

Development 

During  this  phase  the  most  promising  technical 
approach  will  be  converted  into  a,  "stable,  producible,  and 
cost  effective  system  design."  [Ref  12:p.  3-21]  The 
manufacturing  process  will  be  validated  and  contract 
compliance  will  be  determined  through  system  testing.  The 
operational  suitability  of  the  system  will  be  determined 
and  a  production  baseline  will  be  formulated.  [Ref.  12:p. 
3-21] 

h.  Milestone  III:  Production  Approval 

As  with  previous  decision  points,  the  purpose  of 
this  checkpoint  is  to  verify  that  the  exit  criteria 
specified  by  the  Milestone  II  ADM  have  been  met.  The 
results  of  engineering  and  manufacturing  evaluations  must 
support  a  conclusion  that  the  new  system  design  can  be 
efficiently  produced,  is  operationally  acceptable  and 
logistical ly  supportable.  By  this  time,  most  technical  risk 
factors  should  have  been  eliminated.  The  status  of 
program  risk  issues  will  be  contained  in  the  IPS.  The  cost, 
schedule,  and  performance  characteristics  of  the  new  system 
will  be  defined  by  how  well  risk  was  identified  and 
countered.  Successful  completion  of  a  Milestone  III  review 
leads  to  the  authorization  for  full-scale  production  of  the 
new  system.  The  resultant  ADM  will  specify  any  program- 
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specific  exit  criteria  that  must  be  accomplished  during 
Phase  III,  including  refined  program  cost,  schedule,  and 
performance  objectives.  [Ref.  12 :p.  3-24] 

i.  Phase  III:  Production  and  Deployment 

The  goal  of  Phase  III  is  to  field  an  operational 
capability  that  meets  the  MNS.  This  is  achieved  by  a 
stable  design  and  efficient  production  and  support 
processes.  Tlie  system  performance  is  monitored  to  ensure 
compliance  with  MNS  requirements  and  maintain  system 
capability  against  projected  threats.  [Ref.  12 ;p. 3-27] 

j.  Milestone  IV;  Major  Modification 

Approval 

As  required,  the  key  objective  here  is  to 
determine  whether  upgrades  to  existing  systems  are  required, 
prudent,  and  cost-effective  in  light  of  projected  threat 
capabilities  or  emergent  military  requirements.  Technical 
risk  issues  re-enter  at  this  point.  The  technical  approach 
selected  for  a  potential  upgrade  will  affect  not  only  the 
performance  of  the  upgrade  but  also  the  underlying 
capabilities  of  the  host  platform.  The  resultant  ADM  will 
specify  at  what  phase  of  the  acquisition  process  the 
proposed  modification  will  enter  and  approve  the  modified 
acquisition  strategy  and  baseline.  [Ref.  12:p.  3-29] 
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C.  RISK  ASSESSMENT  PROCEDURES 

The  assessment  and  monitoring  of  technical  risk  by  DoD 
acquisition  authorities  is  a  continuous  requirement 
[Ref.  12 ;p.  6-A-3] .  Senior  Department  of  Defense  managers 
recognized  in  the  early  1980s  that  broad  reforms  were 
required  in  the  total  acquisition  mechanism.  As  a  result. 
Deputy  Secretary  of  Defense  Frank  Carlucci  issued  thirty,  two 
initiatives  in  1981.  The  initiatives  were  aimed  at 
improving  management  control  and  the  efficiency  of  the 
acquisition  process  [Ref.  57:p.  47].  Initiative  11  was 
directed  at  improving  the  manner  in  which  quantitative 
program  risk  was  identified  and  its  magnitude  estimated 
by  Program  Managers  [Ref.  8:p.  33].  As  a  result,  DoD 
Instruction  4245. 7-M,  "Transition  from  Development  to 
Production,"  was  issued. 

1.  DoD  Instruction  4245. 7-M 

A  review,  conducted  by  the  Defense  Science 
Board,  on  acquisition  program  viability  was  conducted  and 
they  concluded  that  there  is,  "no  structural  mechanism  that 
can  articulate  with  any  degree  of  certainty  the  risk 
associated  with  the  engineering  and  manufacturing  elements 
of  the  weapon  system  acquisition  process."  [Ref.  14: p.  1-3] 
Inadequate  understanding  of  technical  risk  factors  was 
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determined  to  be  a  major  source  of  risk  in  itself  [Ref.  14: 
P.9--8].  Currently,  DODI  5000.2  PART  5,  Section  B  addresses 
Risk  Management  and  specifically  calls  out  Program  Manager 
use  of  DoD  4245. 7-M  to  identify  areas  of  program  risk. 

DoD  Instruction  4245. 7-M  segregates  the 
acquisition  process  into  a  rational  grouping  of  industrial 
design  problems  covering  design,  testing,  production, 
facilities,  logistics,  and  management  [Ref.  14:p.  1-8] . 
Technical  risk  is  explicitly  identified  and  a  template  was 
ci-'eated  to  aid  program  managers  and  senior  decision  makers 
in  making  better  use  of  the  technical  risk  information 
available  to  them  [Ref.  14:p.  9-8]. 

ITie  instruction  does  not  specifically  address 
techniques  for  estimating  the  likelihood  of  adverse 
consequences,  nor  give  guidance  on  how  decision-makers 
should  rank  the  riskiness  of  various  alternatives.  What  is 
provided  however,  is  an  outline  for  program  management. 

This  outline  calls  for  the  development  of  a  system  that 
provides  early  identification  of  technical  risk  factors, 
instantaneous  assessment  of  program  status,  and  early 
indications  of  potential  success  or  failure.  Guidance  on 
the  development  of  such  a  risk  assessment  and  monitoring 
system  is  absent.  Program  Managers  are  apparently  left  to 
their  own  initiative  [Ref.  14:p.  9-9]. 
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2.  The  Legacy  of  Carlucci  Initiative  11 

The  commitment  of  senior  DoD  acquisition 
officials  to  the  Carlucci  Initiatives  did  not  filter  down. 

A  study  conducted  by  the  General  Accounting  Office  (GAO)  in 
1986  concluded  that  DoD  had,  "not  carried  through  with  its 
action  plans  on  most  of  the  Carlucci  initiatives."  [Ref.  57: 
p.  48]  An  additional  GAO  study  of  technical  risk  assessment 
concluded,  "The  net  effect  of  Initiative  11  on  technical 
risk  assessment  procedures  has  thus  far  been  negligible." 
[Ref.  8:p.  33] 

a.  GAO  Findings 

The  GAO  study  cited  above  surveyed 
25  major  program  offices  from  all  three  services. 
Standardization  of  assessment  techniques  was  completely 
lacking.  Additionally,  there  was  limited  direction  from  the 
appropriate  service  acquisition  authorities  concerning  how 
risks  should  be  ranked.  Subsequent  GAO  investigations  of 
major  programs  has  revealed  no  change  in  DoD  technical  risk 
assessment  practice  [Refs.  9,  10,  11].  Both  DoD  Instruction 
5000.2  and  DoD  Manual  5000. 2-M  [Refs.  12,  13]  are  devoid  of 
the  explicit  guidance  recommended  by  the  GAO  in  1986  [Ref. 
8;p.  77] . 

b.  Risk  Compensation 

Compounding  DoD  technical  risk  problems 
is  the  apparent  practice  of  examining  the  technical  risk  of 
various  programs  as  discrete  events.  While  some  commercial 
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compensation  options  are  unavailable  to  DoD  management, 
e.g. ,  hedging  or  other  financial  strategies,  there  is  no 
direct  mechanism  for  combining  risk  information  into  a 
department-wide  risk  profile.  Risk  compensation  and 
reduction  is  tackled  at  the  program  level  by  individual 
program  managers.  The  absence  of  a  standardized  technical 
risk  quantification  scheme  hampers  senior  management's 
attempts  to  establish  and  manage  a  departmental  risk 
portfolio.  Lacking  quantifiable  and  comparable  technical 
risk  data  and  common  baseline,  technology  trade-off 
decisions  are  difficult  [Ref  8:p.  51]. 

3.  Corporate  Information  Management 

initiative  (CIM  Initiative) 

Paul  Strassman,  Director  of  Defense 
Information,  is  attempting  to  implement  CIM  within  the  DoD. 
CIM  is  built  upon  a  "business  case"  approach  to  analyze  the, 
"potential  costs-saving  Alternatives  for  DoD  information 
management."  [Ref.  54:p.  1]  A  key  feature  of  the  CIM 
initiative  Is  the  use  of  the  quantitative  risk  assessment 
techniques  he  developed  in  his  book.  The  Business  Value  of 
Computers:  An  Executive's  Guide  [Ref.  52].  Strassman 's 
goal  is  to  conduct  a  Risk  Adjusted  Cash  Flow  analysis  of  new 
information  technology  acquisitions  using  Monte  Carlo 
simulations.  The  Institute  for  Defense  Analysis  created  a 
spreadsheet  software  package  that  will  run  a  user-specified 
number  of  trials  with  user-entered  probability  values.  The 
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output  of  the  package  is  a  series  of  graphs  and  tables 
depicting  the  relative  costs  and  benefits  accruing  from 
different  information  technology  alternatives.  An  Expected 
Value  comparison  is  the  decision  criterion.  While  the 
program  accepts  user-defined  probability  values  for  the 
likelihood  of  achieving  various  outcomes,  individual 
judgement  is  relied  upon  for  probability  generation  [Ref. 
54:pp.  19-23]. 

D .  SUMMARY 

This  chapter  highlighted  the  key  features  of  the 
major  weapon  systems  acquisition  process.  Basic  decision 
criteria  and  considerations  were  presented.  The  functions 
of  the  Defense  Acquisition  Board  (DAB)  and  the  primary 
decision  authorities  were  listed.  The  importance  of 
technical  risk  assessment  to  the  acquisition  process  was 
presented.  The  origin  of  current  directives  addressing 
technical  risk  assessment  in  the  1981  CarTucci  Initiatives 
was  noted.  The  continuing  absence  of  formal  and  explicit 
guidance  on  methods  to  be  used  to  assess  and  rank  the 
technical  risk  associated  with  new  system  acquisitions  was 
also  d i s cussed . 

The  consideration  of  technical  risk  on  a  project-by- 
project  basis  without  specific  guidance  for  quantifying 
technical  risk  robs  senior  decision  makers  of  valuable 
information  necessary  to  manage  a  Department  of  Defense-wide 
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technical  risk  portfolio.  Without  quantifiable 
technical  risk  data,  the  overall  impact  of  the  risk 
associated  with  the  technologies  DoD  is  developing  is 
uncertain.  Categorization  of  risks  as  low,  medium,  or  high 
is  left  to  individual  program  managers  and  no  baseline  for 
comparison  exists.  As  Paul  Strassman  has  observed>  "To 
undei^stand  your  risks,  you  need  to  know  how  much  money  you 
could  lose."  [Ref.  52:p.  217] 

Wliile  certain  risk  compensation  strategies  employed  by 
commercial  operations  are  unavailable  to  DoD,  e.g.,  hedging 
and  alternative  financial  markets,  a  portfolio  approach 
based  upon  quantifiable  and  comparable  risk  data  is  still 
applicable.  It  allows  for  a  more  rational  and  systematic 
improvement  of  any  desired  DoD  technical  risk  profile.  The 
overall  Department  of  Defense  technical  risk  position  needs 
to  be  examined  and  explicitly  managed  to  provide  an  overall 
technically  superior  military  force  with  limited  funds. 
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V.  PROPOSED  METHODOLOGY 

A.  BACKGROUND 

Although  the  axioms  of  Utility  Theory  are  logically 
consistent  and  lead  to  coherent  solutions  in  principle, 
significant  problems  remain  with  the  elicitation  of  specific 
utility  values  [Refs.  17,  26,  27] .  As  discussed  in  Chapter 
III,  the  approaches  used  by  commercial  ventures,  many 
decision  makers  are  reluctant  to  use  the  recommended 
techniques  of  standard  devices  of  50:50  reference  gambles, 
probability  wheels  or  urns  full  of  different  colored  balls. 
The  most  strident  reclama  in  DoD ' s  response  to  the  GAO 
report  on  the  Navy's  T-45  program  was  reserved  for  use  of 
the  phrase  "calculated  gamble"  by  the  report's  authors 
[Ref.  9].  Probability  assessment  approaches  are  fraught 
with  considerable  peril  and  often  require  significant 
assistance  from  analysts  who  are  outside  of  the  decision 
making  organization.  Significant  time  and  resources  may 
have  to  be  committed  to  familiarize  a  decision  analysis  team 
with  the  nature  of  the  problem.  Several  iterations 
involving  coherence  and  sensitivity  checks  may  be  required 
to  ensure  a  satisfactory  result.  Finally,  the  results  of 
tlie  analysis  may  require  considerable  distillation  and 
explanation.  Previous  systems  hailed  as  possible  solutions 
to  various  risk  problems  have  frequently  drawn  criticism 
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from  program  managers  because  of  these  factors  [Ref.  60] . 

The  expertise  often  resides  outside  the  program  office, 
requiring  the  program  manager  to  rely  upon  systems  he  may 
not  fully  understand  and  personnel  who  are  not  directly 
accountable  to  him. 

1.  Limitations  of  Expert  Opinion 

An  intuitive  approach  to  the  solution  of  any 
complex  problem  is  the  elicitation  of  engineering  or 
scientific  expert  opinion.  While  a  group  of  experts  can 
provide  valuable  insights  into  likely  solution  methods  and 
their  individual  opinions  codified  into  a  consensus,  their 
estimation  of  the  associated  probabilities  of  success  or 
failure  may  be  tremendously  ambiguous  and  overconfident 
[Ref.  26] .  In  a  1980  study  conducted  for  the  U.S. 

Department  of  Energy,  Salem  et  al .  [Ref.  31]  cite  an 
instance  where  expert  opinion  was  elicited  to  determine  the 
likelihood  of  a  catastrophic  seismic  event  involving  several 
different  nuclear  reactor  installations.  The  seven  experts 
polled  were  from  the  fields  of  civil  engineering,  geology, 
and  geophysics. 

There  (sic)  opinions  as  to  the  probabilities  of  large 
earthquakes  varied  by  as  much  as  four  orders  of 
magnitude  (i.e.,  a  factor  of  10,000)  in  more  than  one 
instance.  Equally  as  interesting  is  the  fact  that 
several  of  the  seven  participants  estimated  their 
uncertainties  at  less  than  a  single  order  of 
magnitude  (two  estimated  their  general  uncertainties 
as  a  factor  of  two  or  less).  (Ref.  31:p.  34] 
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In  an  article  on  competitive  bidding  strategies 
for  the  purchase  of  government-owned  oil  reserves,  it  was 
noted  that  even  though  two  companies  had  worked  closely 
together  on  the  surveying  and  evaluating  of  potential 
reserve  size  and  had  completely  shared  all  the  resulting 
data,  the  subsequent  bids  submitted  for  individual  parcels 
where  dramatically  different  [Ref.  45]. 

2.  Utility  Assessment  Issues 

Assuming  the  decision  maker  is  willing  to 
investigate  a  utility  function  approach  to  his  problem,  then 
his  decision  problem  can  be  reduced  to  a  number  of  smaller 
sub-problems.  This  approach  is  favored  by  various 
engineering  disciplines  and  matches  the  DoD  Work  Break-Down 
Structure  required  during  program  review.  As  discussed  in 
Chapter  II,  the  value  of  utility  function  analysis  lies  in 
its  ability  to  incorporate  preferences  and  risk  perceptions 
in  a  coherent  fashion. 

While  risk-neutral  behavior  is  often  suggested  for 
governmental  operations,  implying  strict  adherence  to 
Expected  Value  maximization,  individual  decision  makers  and 
di.sti  net  governmental  entities,  i.e.,  DoD,  Department  of 
Energy,  Food  and  Drug  Administration,  etc.,  may  be  extremely 
risk  averse  [Refs.  17,  31]  because  of  the  immense 
consequences  of  particular  decisions.  Obtaining  reliable 
Expected  Values  requires  the  use  of  probability 
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distributions  characterizing  a  well  understood  process. 

This  can  be  extremely  difficult  to  obtain  for  some 
"one-of-a-kind'*  problems. 

By  resorting  to  an  analysis  of  his  preferences 
between  two  separate  prospects,  subject  to  some  probability 
distribution,  a  decision  maker’s  risk  function  can  be 
encoded  to  provide  as  much  detail  as  desired.  The 
assessment  method  involves  the  comparison  of  two  prospects 
presented  as  follows: 

X  =  A  guaranteed  payoff 

Payoff  of  value  G  with  probability  p 

Y  - 

Payoff  of  value  L  with  probability  (1  -  p) 

Tlie  values  of  G  and  L  are  not  constrained  to  be  greater  than 
zero.  If  any  three  of  the  four  variables,  X,  G,  L,  and  p, 
are  fixed,  then  the  fourth  can  be  determined.  The  decision 
maker  must  establish  the  effective  end  points  of  the 
problem,  the  maximum  assigned  a  utility  value  of  one  and  the 
minimum  a  value  of  zero.  The  assessment  process  then  seeks 
to  uncover  the  value/utility  pairs  for  the  defined  interval. 
There  are  four  basic  methods  for  eliciting  the  requisite 
information:  Certainty  Equivalence,  Probability  Equivalence, 
Gain  Equivalence,  and  Loss  Equivalence  [Ref.  17]. 


91 


a.  Certainty  Equivalence 

Recall  from  Chapter  II  that  a  Certainty 
Equivalent  (CE)  is  a  guaranteed  payoff  used  to  determine  the 
values  of  gain,  loss,  and  associated  probability  of  an 
opposing  prospect.  Gain,  loss,  and/or  probability  are 
adjusted  by  the  decision  maker  so  that  he  is  indifferent 
between  the  value  of  a  certain  payoff  and  the  expected  value 
of  a  prospect  operating  under  conditions  of  uncertainty. 

In  this  approach,  G,  L,  and  p  are  fixed 
and  the  decision  maker  is  requested  to  provide  an  X,  the 
value  of  the  guaranteed  payoff.  Bunn  states  that  this 
niofhod  fecids  1.0  emphasize  more  risk  averse  behavior  relative 
to  potential  gains  and  more  risk-seeking  behavior  relative 
to  potential  losses  [Ref.  173.  Goodwin  and  Wright  point  out 
that  the  manner  in  which  the  elicitation  question  is  phrased 
substantially  impacts  the  nature  of  the  response  [Ref.  27]. 
Goodwin  and  Wright  use  the  following  example  of  certainty 
equivalent  elicitation  to  underscore  this  point: 

Insurance  Formulation 

A:  You  have  one  chance  out  of  1,000  of  losing  $1,000 
B:  You  can  buy  insurance  for  $10  to  protect  against  this 
loss. 

Gamble  Formulation 

A;  You  stand  one  chance  out  of  1,000  of  losing  $1000 
B:  You  will  lose  $10  with  certainty 
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It  was  discovered  that  81  percent  of  the  subjects  preferred 
option  B  in  the  Insurance  formulation  as  opposed  to  56 
percent  who  preferred  B  when  the  prospects  where  framed  in 
terms  of  a  gamble  [Ref .  27 : p .  81 ] . 

A  further  consideration  with  certainty 
equivalence  occurs  if  the  operable  probability  distribution 
is  skewed.  If  the  decision  problem  involves  only  very  high 
oi-  very  low  probabilities,  the  50:50  construct  should  be 
avoided. 

b.  Probability  Equivalence 

In  similar  fashion,  probability 
equivalence  fixes  the  values  of  X,  G,  and  L  and  requires  the 
decision  maker  to  assess  p.  While  this  tends  to  provide  an  . 
average  risk  attitude,  it  is  subject  to  the  kinds  of 
limitations  discussed  previously  about  human  probability 
assessment  capabilities,  particularly  when  dealing  with  very 
l  oo—  vofy  small  probability  values  or  when  attempting  to 
elicit  responses  for  small  changes  in  likelihood,  e.g., 
moving  from  0.90  to  0.95  [Ref.  27].  Despite  this.  Certainty 
and  Probability  equivalence  are  the  two  more  popular 
techniqnes . 

c.  Gain  Equivalence 

In  the  Gain  Equivalence  method,  value 
for  certainty,  loss,  and  probability  of  gain  are  fixed  and 
the  decision  maker  is  required  to  assess  an  appropriate  gain 
value.  An  issue  with  this  approach,  as  with  Loss 
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Equivalence,  is  the  affect  of  anchoring  bias  as  described  by 
Kahneman  and  Tversky  [Ref.  26].  The  manner  of  incrementing 
the  Gain  value,  as  well  as  the  determination  of  the  initial 
value,  can  affect  the  response. 

d.  Loss  Equivalence 

Loss  Equivalence  follows  the  above 
procedure  with  the  certainty  equivalent,  gain,  and 
[probability  values  now  fixed  and  the  corresponding  potential 
loss  value  assessed  by  the  decision  maker. 

e .  Summary 

As  developed  in  Chapter  II, 

theoielica 1 ly ,  the  utility  function  approach  allows  the 
decision  maker  to  bring  his  perceptions  and  judgements  about 
the  potential  risk  of  an  endeavor  explicitly  into  a  problem 
solution  in  a  coherent  manner.  The  four  techniques 
discussed  above  provide  methods  for  accomplishing  this  in  a 
generic  way  but  the  limitations  of  each  must  be  recognized. 
As  pointed  out  in  Chapter  II,  the  development  of  utility 
theory  arose  through  the  consideration  of  monetary  results. 
Much  subsequent  research  has  been  spent  on  various  means  of 
equating  monetary  and  non-monetary  values  [Refs.  17:pp. 
102-107,  31] .  The  acquisition  decision  maker  faces  a 
similar  problem.  How  should  technology  be  "valued"  and  what 
constitutes  "technical  risk?"  These  questions  will  be 
considered  in  the  next  two  sections. 
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B.  TECHNICAL  RISK 

As  was  pointed  out  in  Chapter  I,  the  force  employment 
strategies  favored  by  the  United  States  rely  upon  the 
fielding  and  maintaining  a  technologically  superior 
military.  Many  resources  are  expended  in  monitoring  the 
technological  capabilities  of  potential  adversaries.  As  DoD 
approaches  the  problem  of  maintaining  a  qualitative 
technical  edge,  an  element  of  uncertainty  is  introduced. 
Solutions  to  force  capability  problems  will  be  selected  that 
ideally  will  provide  the  requisite  technical  performance. 

H  tlio  resultant  system  may  not  possess  the  necessary 

performance  when  finally  fielded.  Any  technical  performance 
h'ss  tlion  tlio  specified  value  may  result  in  a  system  that 
once  fielded,  is  unable  to  meet ‘force  employment  needs  in 
all  intended  tlioaters  of  action.  Alternatively,  the  system 
may  not  possess  the  effective  lifetime  originally  intended 
because  of  failure  to  achieve  the  specified  technical 
performance. 

1.  Technical  Risk  Factors 

Technical  risk  is  affected  by  two  related 
factors:  realized  technological  value  of  the  proposed  system 
V€u:  SUB  ini  ended  performance  and  obsolescence. 

a.  Realized  Technical  Value 

Despite  the  best  of  intentions,  a 
particular  level  of  performance  may  not  be  achieved,  given 
the  chdracterist ics  of  the  technical  approach  followed. 
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Til  is  can  occur  because  the  overall  level  of  technical 
capability  was  insufficient  to  reach  the  goal  or  because 
other  program  constraints  forced  a  reduction  in  the 
resources  available  for  goal  attainment.  As  a  result,  the 
fielded  system,  even  if  developed  within  the  specified  time 
period  may  fall  below  its  performance  goals.  The  risk  in 
this  instance  is  whether  the  nearest  competitor  system  can 
exceed  the  performance  of  the  fielded  system.  While  this 
aspect  of  technical  risk  may  be  mitigated  by  pursuit  of 
parallel  approaches,  at  some  point  the  acquisition  decision 
maker  will  have  to  commit  himself  to  one  system  to  meet 
force  requirements. 

b.  Obsolescence 

The  time  required  to  move  a  system  from 
the  design  phase  to  IOC  will  ultimately  affect  its  useful 
life.  Systems  are  designed  to  provide  a  specific  amount 
per -f ormance  over  a  designated  period.  Tlie  primary 
consideration  being  some  amount  of  qualitative  edge,  i.e., 
Iret!  er-  t])an  llie  nearest  competitor  by  some  specified  amount. 
Useful  system  life  hinges  upon  what  else  is  occurring  in 
related  technological  development  efforts  while  pursuing  a 
particular  approach:  What  is  happening  to  the  state-of-the- 
art?  If  the  project  takes  longer  than  anticipated  to  bring 
to  IOC,  and  even  if  technical  performance  goals  are  met,  the 
useful  life  of  the  system  may  be  significantly  shortened. 
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2.  Technology  Valuation 

Technology  has  no  intrinsic  value.  Its  worth, 
like  njoney,  is  only  in  the  services  it  provides.  Any 
attempt  to  measure  a  technological  value  must  resort  to 
compdi- dl  i  ve  techniques.  Automobiles  are  valued  more  highly 
than  a  horse  and  carriage,  by  some,  because  of  the 
flexibility  in  transportation  and  perceived  ease  of 
ownership.  Likewise,  many  people  prefer  mass  transit 
systems  to  automobiles.  The  value  of  the  automobile  is 
relative  to  the  perceptions  and  environment  of  the 
individual  conducting  the  valuation. 

Similarly,  military  systems  are  valuable  only 
in  relation  to  potentially  competitive  military  systems. 

The  muzzle-loading  rifle  may  have  some  value  to  antique  gun 
collectors  but  its  value  as  a  present-day  weapon  is  nil.  It 
is  non-competitive  with  its  host  of  potential  competitors. 
Given  a  mission  or  goal,  it  should  be  possible  to  state, 
whether  a  particular  system  is  capable  of  functioning  in  a 
marifior  sufficient  to  lead  to  the  accomplishment  of  a  mission 
or  attainment  of  a  goal.  This  implies  the  use  of  judgement 
by  the  individual  or  group  responsible  for  the  mission  or 
goal  to  determine  the  capabilities  of  the  technology  in 
question. 

The  problem  then  becomes  of  one  of 
quant j f Tcation  versus  qualification.  If  scarce  resources 
are  to  be  expended  to  obtain  the  services  of  a  particular 
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piece  of  technology,  it  would  be  worthwhile  to  be  able  to 
state  how  much  better  one  proposed  technological 
implementation  is  than  another.  The  quantification  process 
forces  the  careful  consideration  of  potential  alternatives, 
counter-poised  with  the  likely  environments,  in  which  the 
candidate  technologies  may  be  required  to  operate  and  the 
potential  missions  requiring  support.  The  valuation  method 
adopted  here  is  the  TASCFORM' model,  developed  by  The 
Analytic  Sciences  Corporation  for  the  Director,  Net 
Assessment,  Office  of  the  Secretary  of  Defense,  which  will 
be  explained  in  the  following  section. 

C.  TECHNOLOGY  VALUATION  MODEL 

1.  The  Technology  Valuation  Model 

The  TASCFORM"''"'  model.  Technique  for  Assessing 
Comparative  Force  Modernization,  is  means  of  indexing  the 
technical  performance  characteristics  of  fielded  military 
systems  [Ref.  61].  It  provides  a  non-dimensional  number  or 
figure  of  merit  intended  to  capture  the  multiple  attributes 
associated  with  various  weapons  platforms.  This  approach 
provides  a  decision  maker  with  an  ordinal  scale  which  can  be 
used  to  compare  and  rank  the  technological  value  of 
different  systems.  Additionally,  it  allows  the  decision 
maker  to  observe  how  much  more  technical  performance  one 
system  may  have  when  compared  with  another.  TASCFORM"'"'"'  uses 
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an  additive  and  multiplicative  multi-attribute  approach  with 
"operdt. ional ly-oriented”  subjective  weighting  factors  to 
handle  the  magnitude  and  importance  of  various  attributes. 

The  TAb'CFORM^  approach  incorporates  a  variety 
of  salienL  characteristics  such  as  payload,  range,  speed, 
mobility,  navigation  and  target  acquisition  measures 
[Ref.  61].  The  basic  form  of  the  additive  model  is: 

I  “  klVl  +  +  .  .  .  +knVn 

wticre  I  is  i.lic  overall  platform  index,  ka. ,  .  .  .  ,  km  are 
the  subjec! ively  assigned  weights  and  Vi ,  .  .  .  ,  Vm  are  the 
technology  values  of  the  respective  attributes. 

The  multiplicative  form  is: 

I  =  Va.(ktVl  +  ks>V=.  +  .  .  .  +kmVm> 

This  approach  is  used  when  some  Vi,  a  particular 
technological  variable,  must  be  included  in  the  system. 
Possible  examples  of  this  requirement  would  be  a 
survivability,  range,  or  payload  value. 

2  .  TASCFORMT* Application 

While  TASCFORM^*^  indices  have  been  developed  for 
a  wide  range  of  forces,  for  the  purposes  of  this  thesis,  the 
values  developed  for  tactical  and  ASW  aircraft  were  used. 

The  baseline  for  the  indices  is  the  technology  incorporated 
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in  the  F-4B,  a  circa  1959  aircraft.  Composite  technology 

values  representing  the  synthesis  of  basic  airframe 
performance  (speed,  maneuverability,  range,  and  payload)  and 
imbedded  weapon  system  capabilities  (target  acquisition, 
navigation,  counter-measure  susceptibility,  etc.)  were  used, 
because  this  represents  a  technology  index  relevant  to 
mission  employment. 

3.  Limitations 

The  indices  produced  by  the  TASCFORM"^’’''  model  are 
not,  in  themselves,  predictors  of  the  potential  combat 
success  or  failure  of  the  platforms  under  consideration. 
While  not  scenario  specific,  the  weighting  factors 
incorporated  in  the  sample  reflect  operational  criteria 
relevant  to  a  U.S. -Soviet  engagement  in  Central  Europe. 
Application  of  the  indices  to  other  theaters  involving 
Soviet  or  other  national ity  weapons  would  necessitate 
readjustment  of  the  values.  The  indices  obtained  are 
independent  of  the  likelihood  of  occurrence  of  a  particular 
scenario.  Thus,  while  the  probability  of  a  conflict  between 
U.S.  forces  and  those  of  the  erstwhile  Soviet  Union  may  now 
be  very  small,  such  a  scenario  represents  an  extremely 
challenging  technical  environment.  Cost  data  are  not 
incorporated  in  the  model,  which  facilitates  a  direct 
approach  to  the  assessment  of  technical  risk  unmodified  by 
other  considerations  [Ref.  61]. 
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D.  PROPOSED  TECHNICAL  RISK  ASSESSMENT  METHOD 

Any  method  for  assessing  technical  risk  that  seeks  to 
provide  coherent  solutions  must  be  sensitive  to  both  the 
axiomatic  requirements  of  Utility  Theory  and  the  practical 
issues  of  implementation.  Two  critical  obstacles  to  the 
effective  application  of  Utility  Theory  have  been  the  issue 
of  probability  assessment  and  the  use  of  reference  gambles. 
The  "standard  device"  procedure  is  subject  to  trivial ization 
on  the  part  of  the  decision  maker.  The  following  proposed 

'i 

method  for  assessing  technical  risk,  attempts  to  circumvent 
these  problems  through  the  provision  of  an  explicit 
probability  distribution  and  a  valuation  system  that  keeps 
the  technology  utility  function  assessment  process  confined 
to  issues  of  technical  value. 

1.  Assumptions 

Following  the  work  of  Moses,  Dodson,  and  Knight 
[Refs.  62,  63,  64],  the  following  assumptions  were  made*. 

a.  Technological  value  can  be  quantified 

and  measur’ed. 

b.  The  growth  of  U.S.  military 
Lechiiolog ical  value  over  time  can  be  modeled  using  linear 
regression  techniques. 

c.  A  probability  distribution  can  be 
derived  from  the  resulting  regression  equation  that  will 
allow  an  acquisition  decision  maker  to  determine  his 
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likelihood  of  obtaining  a  system  possessing  some  specified 
lechnological  performance  value  based  upon  the  results  of 
previous  technological  development. 

d.  The  performance  value  of  the  closest 
likely  competitor  technology  can  be  estimated  using  the 
TASCFORM.M  j^^thod  [Ref.  61]. 

e.  Using  the  Certainty  Equivalent  technique 
described  above,  the  acquisition  decision  maker  can  quantify 
l^is  technological  risk  perceptions  directly  and  incorporate 
these  into  a  utility  curve  that  would  assist  him  in 
determining  a  satisfactory  approach  to  meeting  the  mission 
needs  of  military  forces  while  incorporating  a  notion  of 
technical  risk. 

2.*  System  Technology  Sample 

Following  the  work  of  Moses  [Ref.  62],  the 
statistical  model  used  a  sample  of  49  U.S.  Navy  and  Air 
Force  airci''aft  developed  between  1950  and  1979.  The  sample 
was  restricted  to  tactical  and  anti-submarine  fixed  wing 
platforms  for  which  TASCFORM"'"'"'  performance  index  values 
where  available.  Source  data  for  year  of  IOC  and  composite 
technology  values  were  obtained  from  the  U.S.  Military 
Aircraft  Cost  Handbook  [Ref.  65]  which  contained  TASCFORM’^'^ 
composite  technical  values  for  the  aircraft  in  the  sample. 
The  technology  values  contained  in  the  Handbook  were  derived 
using  the  TASCFORM^''’’  methodology  discussed  above  and  derived 
[Ref.  61].  The  Aircraft  System  Performance  (ASP)  values 
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wore  used.  The  ASP  value  adjusts  basic  airframe  measures  of 
payload,  range,  speed,  etc.,  for  mission  requirements  of 
'.  at  gel  acquisition,  navigation,  survivability,  etc.  and  thus 
more  truly  reflect  platform  capabilities  [Ref.  61] .  The 
resultant  values  are  tabulated  in  Table  5-1  by  airframe 
designator,  year  of  IOC,  and  composite  technology  value. 


AIRCRAFT  TECHNOLOGY  MEASURES 
TABLE  5-1 

AIRCRAFT  IOC  YEAR  TASCFORM  COMPOSITE 

(1950-79)  PERFORMANCE  INDEX 


F-89C 

50 

2.46 

F-9F/H 

51 

4.19 

F-89A 

51 

4.05 

F-84F 

51 

5.13 

F-86F 

51 

4.03 

F-86D 

51 

3.68 

F-2C 

51 

3.91 

F-3A/B/C 

52 

9.02 

F-IB/C/M 

52 

5.29 

F-86H 

52 

5.68 

F-IOOA/C 

52 

4.80 

F-llA 

53 

5.80 

A-3A/B 

53 

10.74 

F-102A 

53 

9.71 

F~6A 

53 

7.58 

A-4A/B 

53 

3.93 

F~1E 

54 

5.44 

F-IOIA/B 

54 

13.55 

F~10UD 

54 

5.99 

F-8A/B/C 

55 

8.40 

A-IJ 

55 

3.34 

F-9F 

55 

4.02 

A-IE/G/H 

56 

3.34 

F-104A/B 

56 

6.79 

F-106A/B 

57 

13.05 

F-105B/D 

57 

14.86 

A~4C 

57 

5.45 

F-4A/B 

59 

9.32 

A-6A 

61 

13.83 

A-4E/F 

61 

7.27 

F-4C/D 

62 

10.07 
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AIRCRAFT  TECHNOLOGY  MEASURES 
TABLE  5-1 


AIRCRAFT 

IOC  YEAR 
(1950-79) 

TASCFORM  COMPOSITE 
PERFORMANCE  INDEX 

P-3C 

65 

30.33 

A-7A/B 

65 

12.10 

F-lllA 

65 

18.46 

F-4E 

66 

13.96 

F-lllB 

66 

24.81 

F-4J 

66 

13.39 

A-7E 

68 

19.77 

A-7D 

68 

16.17 

F-lllD 

68 

24.39 

S-3A 

69 

20.21 

F-lllF 

70 

31.01 

A~4M 

70 

8.52 

A-6E 

70 

22.40 

F-14A 

71 

31.51 

F-15A 

73 

16.14 

A-lOA 

75 

12.12 

F-16A 

78 

15.69 

F/A-18A 

79 

25.42 

3.  Regression  Equation 

The  regression  model  was  hypothesized  to  be 

liiifidr  of  the  form: 


FLYTECH  =  CONSTANT  +  A  *  YEAR  +  e. 


where  "FLYTECH"  is  the  TASCFORM'"''"-derived  composite  ASP 
technology  value,  "YEAR"  the  year  of  platform  IOC,  "A"  a 
regression  coefficient,  and  "e"  any  residual  error  not 
explained  by  the  model.  The  regression  was  run  using  the 
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student  Version  of  MINITAB,  Version  1.1.  Regression 
coefficients  are  provided  below.  Regression  results  are 
presented  in  Table  5-2: 

FLYTECH  =  -32.493  +  0.73724  *  YEAR 

REGRESSION  RESULTS 
TABLE  5-2 

Pi  edict  or  Coefficient  Standard  t-ratio  p 

Deviation 

Conslcint  -32.493  5.507  -5.90  0.000 

year  0.73724  0.09112  8.09  0.000 

s  -  5.228  R^^  =  58.2%  R^(adj)  =  57.3% 

Analysis  ol  Variance 

SOURCE  DF  SS  MS  F  p 

Royreooion  1  3788.9  1788.9  65.46  0.000 

Error  47  1284.4  27.3 

Toi  a  i  48  3U73 . 3 

The  regression  coefficients  vary  slightly  from  those  Moses 
obtained.  [Ref.  62]  because  of  the  inclusion  of  P-3C  Orion 
and  S-3A  Viking  platforms. 

4.  Regression  Goodness  of  Fit 

The  proposed  regression  model  adequately 
(jooct  iliic.i  Die  dii'craft  technology  data.  The  model  is 
significant  at  the  p  “  0.000  level.  Five  different  plots  of 
the  resultant  regression  data  were  constructed.,  as 
recommended  by  Devore  [Ref.  25:pp.  498-503],  to  assist  in 
the  analysis  of  the  regression's  "goodness-of-f it . "  Figure 
5—1  displays  a  plot  of  the  regression  line  and  the  actual 
technology  values  over  time.  Figures  5-2  and  5-3  are  plots 


105 


or  !Jic  oLdiidcirdizod  residuals.  An  analysis  of  the  data 
contained  in  these  figures  shows  that  the  distribution  of 
the  residuals  appears  to  be  random  and  most  of  the  points 
fall  with  plus/minus  two  standard  deviations  of  the  expected 
residual  value  of  zero.  Figure  5-4  displays  actual  versus 
estimated  technology  values.  The  graph  corroborates  the  Re¬ 
value  that  time  in  the  form  of  year  of  IOC  is  a  reasonable 
explanatory  variable  for  technical  value.  Figure  5-5,  a 
normal  probability  plot  of  the  standardized  residuals,  shows 
that  the  residuals  follow  a  straight  line  and  are  contained 
within  the  inLeival  (-2,  2) .  This  tends  to  confirm  the 
assumption  that  the  error  term  is  normally  distributed. 
Figure  5-4  is  a  plot  of  estimated  versus  actual  technology 
values.  Figure  5-5  is  a  normal  probability  plot  of  the 
slandardized  residuals.  A  listing  of  fitted  values, 
residuals,  standardized  residuals  and  normal  probability 
scores  is  contained  in  Appendix  A. 
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TECHNOLOGY  MEASURES 

PLOTTED  OVER  TIME 


TECHNOLOGY  MEASURE 


S«nw  A  S«riw  B 

t990<-79 


Figure  5-1  Plot  of  Technology  Indices  over  Time 


v 
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STANDARD  RESIDUALS 

ACTUAL  vs  YEAR 


STANDARD  RESIDUAL 


Figure  5-2  Plot  of  Standard  Residuals  vs  Year 


STANDARD  RESIDUALS 

ACTUAL  vs  ESTIMATED 


STANDARD  RESIDUALS 


*  S«ries  A 


Figure  5-3  Plot  of  Standard  Residuals  vs  Fitted 
Technology  Indices 
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NORMAL  PROBABILITY  PLOT 


STANDARD  RESIDUALS 


Following  a  method  suggested  by  Montgomery 
[Ref.  66],  a  check  was  made  of  the  standardized  residuals  to 
detemnine  if  there  is  a  significant  difference  in  the 
behavior  of  the  positive  residuals  as  compared  to  the 
negative  residuals.  An  F-test  with  19  and  30  degrees  of 
freedom  was  constructed  as  follows:  The  the  sum  of  the 
variances  for  the  19  positive  residuals  was  divided  by  the 
sum  of  the  30  negative  residual  values.  The  null  hypothesis 
was  that  this  ratio  should  be  equal  to  one: 

Ho:  s=^(  i-f )  =  1  where  s^(i  +  )  are  positive  residuals 

S'"(i-)  and  s=-(i-)  are  negative  residuals 


Hi  :  3^  (  i -f )  /=  1 

S  '"  (  i  - ) 

The  results  are  provided  in  Table  5-3. 

ANALYSIS  OF  VARIANCE  RESULTS 
TABLE  5-3 

s''Mi  +  )  -  29.1547  s-Hi-)  =  20.3376 

s  '-'"  (  i  + )  =  1.43354 
s^(i~) 

F  =  1.43354  p  =  0.184 

The  test  result  indicates  that  the  sums  of  the 
respective  variances  are  the  same  at  the  p  =  0.184  level. 
The  differences  in  the  sums  of  the  respective  variances  be 
partially  explained  by  the  large  effect  of  the  F-14A  and 
F-lllF  technology  indices.  These  values  in  particular  are 
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greater  than  two  standard  deviations  away  from  the  expected 
technology  value  for  the  year  of  IOC. 

5.  Technical  Advance 

Obtaining  the  services  of  a  technology  that 
performs  better  than  the  state  of  the  art  is  an  essential 
driver  in  the  DoD  acquisition  process.  Not  only  does  this 
piovidf-^  a  potentially  large  quantitative  edge  over  likely 
competitor  systems  at  the  time  of  IOC,  it  also  tends  to 
longllieii  Llie  use  life  of  the  system,  in  the  presence  of 
steadily  progressing  technological  performance. 

An  estimate  of  technology  performance  value 
different  from  the  state  of  the  art,  for  a  given  year,  can 
be  obtained  by  taking  the  difference  between  the  actual  and 
estimated  technology  values: 

ADVANCE  =  FLYTECH  ~  SOA, 

where  "FLYTECH"  is  the  TASCFORM-i-m  composite  technology  index 
and  "SOA"  is  the  state-of-the-art  technology  index  based 
upon  year  of  IOC  estimated  by  the  regression  equation.  A 
l>osii. ive  Advance  Value  indicates  a  system  that  possessed 
better  than  state  of  the  art  technology,  while  a  negative 
value  indicates  a  system  that  was  below  the  SOA  at  time  of 
IOC.  The  more  positive  the  Advance  Value,  the  greater  the 
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'oystem'£5  qualitative  edge  over  systems  that  are 

state-of-the-art.  Table  5-4  provides  a  listing  of  Advance 

Values  by  aircraft  designator. 

TECHNOLOGY  VALUES:  MEASURED  VERSUS  ESTIMATED 

TABLE  5-4 

AIRCRAFT  IOC  TASCFORM  CALCULATED  ADVANCE 

YEAR  COMPOSITE  INDEX 


(1950-79) 

INDEX 

F-89C 

50 

,2. 

,46 

4. 

35 

-1. 

89 

F-9F/H 

51 

4. 

19 

5. 

09 

-0. 

90 

F~89A 

51 

4. 

,05 

5. 

,09 

-1. 

,04 

F-84F 

51 

5. 

13 

5. 

09 

0. 

04 

F-86F 

51 

4. 

,03 

5. 

,09 

-1  . 

,06 

F-86D 

51 

3. 

68 

5. 

09 

-1 . 

,41 

F-2C 

51 

3. 

,91 

5. 

.09 

-1 . 

,18 

F-3A/B/C 

52 

9. 

02 

5. 

82 

3. 

,20 

F-IB/C/M 

52 

5. 

.29 

5. 

,82 

-0. 

,53 

F-86H 

52 

5. 

68 

5. 

82 

-0. 

,  14 

F-IOOA/C 

52 

4. 

,80 

5. 

,82 

-1 , 

,02 

F-llA 

53 

5. 

80 

6. 

56 

-0. 

,76 

A-3A/B 

53 

10, 

.74 

6. 

.56 

4, 

.18 

F-102A 

53 

9. 

71 

6. 

56 

3. 

,  15 

F-6A 

53 

7, 

,58 

6, 

.56 

1  , 

.02 

A-4A/B 

53 

3. 

93 

6. 

,56 

-2. 

.63 

F-IE 

54 

5, 

.44 

7, 

,30 

-1  , 

.86 

F-IOIA/B 

54 

13. 

,55 

7. 

,30 

6. 

,25 

F-IOOD 

54 

5, 

.99 

7, 

.30 

-1  , 

.31 

F-8A/B/C 

55 

8. 

,40 

8. 

,03 

0, 

,37 

A-IJ 

55 

3, 

.34 

8, 

.03 

-4 

.69 

F-9F 

55 

4. 

,02 

8. 

,03 

-4, 

.01 

A~1E/G/H 

56 

3, 

.34 

8, 

.77 

-5, 

.43 

F-104A/B 

56 

6. 

,79 

8. 

,77 

-1 , 

.98 

F-106A/B 

57 

13 

.05 

9, 

.51 

3 

.54 

F-105B/D 

57 

14, 

.86 

9, 

.51 

5, 

.35 

A-~4C 

57 

5 

.45 

9 

.51 

-4 

.06 

F-4A/B 

59 

9, 

,32 

10, 

.98 

-1 , 

.66 

A-6A 

61 

13 

.83 

12, 

.46 

1 

.37 

A-4E/F 

61 

7, 

.27 

12, 

.46 

-5, 

.19 

F-4C/D 

62 

10 

.07 

13 

.  19 

-3 

.12 

P~3C 

65 

30, 

.33 

15, 

.41 

14, 

.93 

A~7A/B 

65 

12 

.10 

15 

.41 

-3 

.31 

F-lllA 

65 

18, 

.46 

15, 

.41 

3, 

.06 

F-4E 

66 

13 

.96 

16 

.14 

-2 

.18 

F-lllB 

66 

24, 

.81 

16, 

.14 

8 

.67 

F-4J 

66 

13 

.39 

16 

.14 

-2 

.75 
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TECHNOLOGY  VALUES:  MEASURED  VERSUS  ESTIMATED 

TABLE  5-4 


AIRCRAB'l' 

IOC 

TASCFORM 

CALCULATED 

ADVANCE 

YEAR 

COMPOSITE 

INDEX 

(1950-79) 

INDEX 

A-7E 

68 

19.77 

17.62 

2.15 

A-7D 

68 

16.17 

17.62 

-1.45 

F-lllD 

68 

24.39 

17.62 

6.77 

S-3A 

69 

20.21 

18.35 

1.86 

F-lllF 

70 

31.01 

19.09 

11.92 

A-4M 

70 

8.52 

19.09 

-10.57 

A-6E 

70 

22.40 

19.09 

3.31 

F-14A 

71 

31.51 

19.83 

11.68 

F-15A 

73 

16.14 

21.30 

-5.16 

A-lOA 

75 

12.12 

22.78 

-10.66 

F-16A 

78 

15.69 

24.99 

-9.30 

F/A  -18A 

79 

25.42 

25.72 

-0.30 

The  Advance  Value 

is  calculated 

to  provide  a 

comparative 

technology  value 

in  determining 

the  technology  utility 

function 

It  is 

not  the  absolute  measure  of 

the  index  that 

is  impartdiit ,  but  rather  the  comparative  difference  between 
alternatives  and  likely  competitors  as  viewed  over  time  that 
provides  a  more  meaningful  measure  of  a  system's  technical 
advantages.  An  example  of  the  value  of  a  relatively  large 
Advance  Value  can  be  seen  in  the  TASCFORM"'"''^  F-14A 
performance  value  compared  with  "SOA"  at  time  of  IOC. 

The  11.68  1971  IOC  Advance  Value  is  greater  than 
two  standard  deviations  above  the  regression  estimate.  The 
state-of-the-art  did  not  achieve  this  value  until 
approximately  1987.  The  attainment  of  this  Advance  Value 
provided  for  exceptional  platform  longevity.  On  the  other 
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hand,  the  A-6E  was  characterized  by  an  Advance  Value  of  only 
3.31  at  the  1971  IOC,  necessitating  efforts  in  the  early 
1980s  to  obtain  a  successor  system  . 

E.  TECHNICAL  RISK  AND  UTILITY  DETERMINATION 

The  Advance  Value  information  generated  by  the 
regression  equation  can  now  be  combined  with  the  Certainty 
Equivalent  utility  assessment  technique,  described  above,  to 
create  a  utility  function  that  incorporates  the  decision 
maker's  risk  perception.  The  necessary  steps  are  presented 
below. 

1.  Functional  Limits 

Functional  limits  define  the  interval  on  which 
the  utility  function  will  operate.  These  are  necessarily 
judgemental,  but  can  be  specified.  The  functional  limits 
dofirie  the  space  in  which  the  design  engineers  will  work  to 
create  a  system  that  meets  military  mission  needs.  The 
maximum  value  is  scaled  as  a  utility  value  of  one  and  the 
minimum  acceptable  value  is  assigned  a  utility  value  of 
zero . 

2.  Utility  Elicitation 

Tlie  proposed  method  uses  the  certainty 
equivalence  technique,  posed  in  the  form  of  an  "insurance" 
premium,  and  nests  the  responses  working  inward  from  the 
functional  limits.  Initial  minimum  and  maximum  values  are 
determined  from  the  DoD  Mission  Needs  statement  as  discussed 
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in  Chapter  IV.  This  document  reflects  the  requirements  of 
Unified  and  Specified  Commanders,  Military  Departments,  the 
Office  of  the  Secretary  of  Defense,  or  the  Joint  Staff  to 
meet  national  military  objectives  and  or  counter  current 
threat  capahT 1 i ti es  [Pef.  12].  These  values  reflect  the 
desired  and  minimum  performance  requirements  any  candidate 
system  must  possess  to  meet  the  requirements  of  the  entities 
listed  above.  Subsequent  interval  endpoints  are  obtained 
during  the  utility  elicitation  process. 

a.  Certainty  Equivalence 

A  certainty  equivalence  form,  with 
probability  equal  to  0.5,  is  appropriate  in  this  case.  The 
sample  Advance  Value  distribution  appears  to  have  a 
symmetric,  normal  shape  [Refs.  1*7,  27].  This  approach  also 
abides  by  the  requirement  that  the  certainty  equivalent,  Z, 
lie  within  the  interval  X  <  Z  <  Y,  where  X  is  the  functional 
minimum  and  Y  the  functional  maximum  for  the  interval  under 
consideration. 

b.  Functional  Limit  Determination 

Based  upon  the  analysis  of  mission  needs 
and  estimated  performance  of  likely  competitor  systems,  the 
initial  minimum  acceptable  Advance  Value  for  the  projected 
year  of  IOC  can  be  specified  from  the  TASCFORM'^'^  process  and 
the  SOA  regression  calculations.  The  functional  maximum  can 
be  similarly  determined  from  the  regression  results  using 
either  a  plus  three  standard  deviation  value  from  the 
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regression  estimated  SOA  or  some  increment  to  immediate 
predecessor  technology  performance.  Subsequent  sub¬ 
intervals  will  result  from  the  iterative  utility  assignment 
process . 

3.  Example 

Figure  5-6  shows  a  representative  plot  of 
U.S.,  series  A,  and  Soviet  tactical  air,  series  B, 
performance  indices  derived  from  the  TASCFORM'^'*’  method  as  a 
function  of  time.  As  can  be  seen,  U.S.  platforms  enjoyed 
an  average  advantage  of  about  four  index  units  when  compared 
with  competitor  Soviet  platforms.  Soviet  technical 
capabilities  are  used  for  comparison  purposes  as  these 
represent  systems  widely  used  throughout  the  world  and  pose 
a  significant  military  challenge  to  U.S.  forces  even  if  not 
actually  employed  by  Soviet  forces. 


AVERAGE  PERFORMANCE  INDEX 

FIXED  WING  TACTICAL  AIRCRAFT 


AVCMK  PCm)RMM4Ce  MDEX 


Figure  5-6  U.S.  and  Soviet  Technology  Indices 
Over  Time 
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The  following  is  an  example  of  how  the  proposed 
utility  assessment  process  -would  work.  The  utilities 
cujslgned  to  tlie  various  technology  performance  values  are 
those  of  this  researcher. 

a.  Functional  Limit  Determination 
Suppose  the  overall  force  structure 

requires  that  any  new  air  combat  system  possesses  a  minimum 
technical  performance  value  of  at  least  2.5  index  units 
above  an  anticipated  Soviet  competitor  at  time  of  IOC.  The 
lesultant  performance  index  specifies  the  utility  minimum 
and  is  assigned  a  value  of  zero.  For  this  example,  the 
minimum  Advance  Value  will  be  specified  as  -4.  The  initial 
functional  Advance  Value  maximum  will  be  specified  at  16. 

Th :  i;'  assigned  a  utility  value  of  one. 

b.  Certainty  Equivalence  Assessment 

The  initial  prospect  is  framed  in  the 

following  manner: 

A.  Obtain  a  platform  with  an  Advance  Value  of  16  with 
probability  of  0.50 

B.  Obtain  a  platform  with  Advance  Value  of  -4  with 
probability  of  0.50 

The  expected  value  of  this  prospect  has  an  Advance  Value  of 

6. 

C.  Specify  the  Advance  value  you  would  accept  to  protect 
against  obtaining  a  system  with  an  Advance  Value 

of  -4 . 
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The  answer  to  C  provides  the  Advance  Value  corresponding  to 
a  utility  value  of  0.5.  The  two  new  sub-intervals , 

(C,  16)  and  (-4,  C) ,  are  subsequently  decomposed  into  their 
respective  utility  values  by  means  of  the  nesting  technique 
discussed  previously.  Examples  of  possible  results  are 
presented  in  Table  5-5  and  Figure  5-7. 

TECHNOLOGY  UTILITY  ASSESSMENT  EXAMPLE 
TABLE  5-5 


ADVANCE 

U( ADVANCE) 

-4.00 

0.00 

0.00 

0.125 

2.00 

0.25 

4.00 

0.50 

8.00 

0.75 

14.00 

0.875 

16.00 

1.00 

TECHNOLOGY  UTILITY  FUNCTION 


TECHNOLOCr  UTIUTY 


—  Series  A 


Figure  5-7  Example  Utility  Determinatoin 
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c.  Analysis 

The  utility  curve  in  Figure  5-7  displays 
risk-uceking  behavior  on  the  interval  (-4,  4),  risk  averse 
behavior  over  the  interval  (4,  10),  and  risk-seeking  on  the 
interval  (10,  16),  The  results  are  consistent,  in  general, 
with  the  axioms  of  Utility  Theory,  as  there  is  no 
requirement  to  display  a  constant  risk  behavior.  The 
combination  of  risk  behaviors  over  the  sub-interval 
precludes  recourse  to  Mean  -  Variance  analysis.  The  next 
step  in  the  analysis  would  be  to  closely  examine  the 
results,  particularly  in  the  region  of  inflection  points,  to 
ensure  that  they  truly  reflect  the  decision  maker's 
prefer-ences  and  sensitivities. 

4.  Utility  Function  Application 

Once  the  function  has  been  determined  and  the 
decision  maker  is  confident  that  it  captures  his 
preferences,  probabilities  determined  from  the  regression 
can  applied.  The  resultant  Expected  Utility  can  then  be 
used  to  select  candidate  system  technologies  for  goal 
accomplishment.  Each  candidate  would  possess  some  Advance 
Value  whose  likelihood  of  success  can  be  ascertained  by  the 
use  of  the  normal  probability  distribution  associated  with 
the  SOA  regression.  Given  a  particular  Advance  Value,  the 
probability  of  achieving  that  value  or  greater  can  be 
directly  calculated.  This  probability,  when  multiplied  by 
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the  corresponding  utility  value,  gives  an  Expected  Utility 
value  (EU) ,  which  can  be  subsequently  ranked  against  other 
candidates. 

F.  UTILITY  FUNCTION  ALTERNATIVE 

Although  the  Utility  Function  approach  presented  above 
will  capture  the  individual  decision  maker's  risk 
pei'cept ions ,  it  does  require  careful  iteration.  Individual 
sensitivities  in  regions  of  curvature  changes  must  be 

oughl  y  examined.  An  additional  consideration  is  the 
organizational  environment  in  which  the  acquisition  decision 
18  made.  An  allernative  approach  is  to  employ  the  Extended 
Pearsoii-Tukey  (EP-T)  method. 

1.  The  Extended  Pearson-Tukey  (EP-T)  Method 

The  EP-T  method,  as  described  by  Goodwin  and 
Wr  iylit,  [Ref  .  27]  has  been  found  useful  in  a  variety  of 
continuous  distribution  cases.  It  breaks  the  ranking  scale 
into  three  segments,  high,  medium,  and  low.  The  three 
categories  requiring  decision  maker  estimation  correspond 
to: 

a.  Value  which  has  a  95%  chance  of  being 
exceeded  (Low) .  This  value  is  assigned  a  probability  of 
0.185. 

b.  Value  which  has  a  50^  chance  of  being 

m 

exceeded  (Medium).  This  is  assigned  a  probability  of  0.63. 
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c.  Value  which  has  only  a  5^  chance  of 
being  exceeded  (High).  This  is  assigned  a  probability  of 
0.185. 

The  probabilities  reflect  the  symmetrical  nature  of  the 
implied  distribution  and  sum  to  one.  The  specific  values  in 
this  case  can  be  obtained  by  analyzing  the  regression 
results  in  terms  of  the  normal  distribution  of  Advance 
Values. 

a.  Example 

Using  the  results  of  the  regression 
analysis,  a  possible  risk-assessment  approach  using  the  EP-T 
method  would  be  conducted  as  follows:  Since  the  residuals 
appear  to  belong  to  a  normal  distribution,  z-values  for  the 
respective  0.95,  0.5,  and  0.05  technology  values  can  be 
obtained  from  a  standard  normal  table.  The  standard 
deviation  value  of  5.288,  obtained  from  the  regression,  is 
then  used  to  arrive  at  specific  technology  values.  The 
results  are  summarized  in  Table  5-6. 

EP-T  METHOD  EXAMPLE 
TABLE  5-6 


%  Chance  of  Exceeding  z-value 
95  -1.645 

50  0 

05  1 . 645 

As  can  be  seen  from  Table  5-4,  30  of 
sample  would  be  considered  low  risk. 


Advance  Value  Risk 
-8 . 6  Low 

0  Medium 

8.6  High 


the  49  aircraft  in  the 
15  would  rank  as 
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medium  risk,  and  four  would  be  high  risk  cases.  Expected 
Value  (EV)  methods,  discussed  in  Chapter  II,  are  then 
applied  to  select  the  candidate  systems  which  provide  the 
highest  EV.  This  may  provide  a  faster  solution  to  the 
problem.  The  disadvantage  lies  within  the  probabilities 
assigned  to  the  three  levels.  As  pointed  out  in  the 
discussion  of  Utility  Theory,  risk  perception  is 
intrinsical ly  personal.  While  organizationally,  critical 
probability  levels  can  be  assigned  by  fiat,  these  may  tend 
to  represent  the  perceptions  of  senior  personnel,  not  the 
entire 

organization. 

G.  SUMMARY 

This  chapter  began  by  highlighting  some  of  the  concerns 
and  probloiiis  associated  with  obtaining  the  probability 
values  necessary  for  the  Utility  Assessment  approach. 
Re'course  1  o  exper'L  op^iiiion  was  sliown  to  be  inadequate  for 
this  task.  The  four  basic  methods  of  Utility  Assessment, 
obdbi  1  i  Ly  Equivalence,  Gain  Equivalence,  Loss  Equivalence, 
and  Certainty  Equivalence,  were  discussed  and  their 
resi>eci  ive  strengths  and  weaknesses  noted.  Two  major 
factors  affecting  technical  risk,  realized  technical  value 
and  obsolescence,  were  explained.  A  means  of  measuring 
technology  on  an  ordinal  scale  was  introduced.  The 
TASCFORMT' •'''  technology  valuation  model  was  presented  and 
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discussed  as  a  means  of  measuring  technology  for  subsequent 
utility  assessment  determination.  A  regression  model  of 
technology  over  time  was  constructed  using  TASCFORM'^'^  values 
for  a  sample  of  49  U.S.  Navy  and  Air  Force  aircraft  spanning 
the  period  1950-1979.  The  purpose  of  this  model  was 
twofold:  To  obtain  a  probability  distribution  that 
characterized  the  attainment  of  technology  values  unaffected 
by  tlie  individual  biases  described  by  Tversky  and  Kahnemann, 
and  to  obtain  a  measure  of  technological  advantage,  an 
“Advance  Value",  that  could  be  used  to  construct  an  utility 
function  for  varying  technology  values.  An  Utility 
Assr-ssiiient  technique,  using  Certainty  Equivalence,  was 
proposed  and  an  example  of  a  possible  risk-assessment 
scenario  was  put  forward.  The  notion  of  Expected  Utility 
using  the  probabilities  determined  from  the  regression  model 
was  discussed.  An  alternative  means  of  risk  assessment,  the 
Extended  Pearson-Tukey  method,  was  discussed,  and  the 
example  reworked  using  this  approach.  Problems  associated 
with  both  methods  were  discussed. 
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VI. 


CONCLUSIONS  AND  RECOMMENDATIONS 


A.  CONCLUSIONS 

1.  Technology  can  be  valued  and  measured  using  an 
ordinal  scale. 

The  technological  characteristics  of  a  weapon  system 
can  be  assigned  numerical  values  in  a  coherent  fashion. 

These  values  can  be  subsequently  used  to  order  alternative 
systems  from  most  capable  to  least  capable.  The  numerical 
differences  between  alternatives  can  provide  an  indication 
of  how  much  better  one  alternative  is  than  another. 

2.  The  increase  in  technological  state-of-the-art  over 
time  can  be  explained  by  a  linear  regression  as  a  first 
approximation. 

As  organizations  become  more  familiar  with 
technological  requirements  and  capabilities,  the 
technological  value  of  new  systems  increases  over  time. 

This  value  can  be  split  into  state-of-the-art  and 
technological  advance  components.  The  projected 
state-of-the-art  can  be  estimated  via  a  linear  regression. 
Technological  advance  represents  capability  over  and  above 
the  state-of-the-art  at  the  time  the  system  is  fielded. 
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3.  The  probability  of  achieving  a  particular  level  of 

technological  value  can  be  estimated. 

The  probability  distribution  characterizing  the 
regression  residuals  can  be  used  to  estimate  the  likelihood 
of  achieving  a  particular  technological  value  at  a  specific 
moment  in  time.  This  removes  a  serious  problem  in  risk 
assessment:  estimating  the  probability  of  obtaining  a 
certain  level  of  technical  performance.  Expert  judgement 
can  be  focused  on  determining  the  technical  requirements  of 
a  system  and  valuing  the  technology  inherent  in  a  new 
weapons  system. 

4.  Utility  functions  for  technology  can  be  created. 

A  utility  function  for  technology  values 
characterizing  a  range  of  system  alternatives  can  be 
created,  representing  the  risk  preference  behavior  of  an 
individual.  The  utility  function  can  be  found  using  the 
Certainty  Equivalent  method  for  technology  valuation  systems 
possessing  symmetrically  distributed  residuals, 

5.  The  Extended  Pearson-Tukey  (EP-T)  method  can  be 
applied  to  determine  the  risk  inherent  in  a  range  of 
alternatives. 

The  EP-T  method  can  be  applied  to  technology 
valuation  systems  possessing  symmetrically  distributed 
residuals.  This  approach,  while  less  sensitive  to 
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individual  risk  preferences,  is  much  faster  and  lends  itself 
to  conventional  Expected  Value  comparisons  between  competing 
options. 

B.  RECOMMENDATIONS 

1.  The  Department  of  Defense  should  continue  exploring 
the  utility  of  TASCFORMT ’'''-1  ike  technology  valuation  methods 
for  technical  risk  assessment. 

This  would  focus  expert  knowledge  on  areas  of  value, 
which  would  be  more  amenable  to  expert  judgement,  and  avoid 
estimating  probabilities  by  human  judgement.  Probabilities 
can  be  analytically  determined  for  those  cases  where  a 
linear  regression  approximates  the  growth  of  technological 
value  over  time. 

2.  DoD  should  investigate  the  application  of  the 
Extended  Pearson-Tukey  (EP-T)  method  for  characterizing 
technical  risk  via  a  pilot  program  approach. 

Whiile  the  generation  of  individual  technology 
utility  functions  can  represent  an  individual's  risk 
perception  and  risk  seeking  behavior,  a  Utility  Theory 
approach  may  be  too  cumbersome  to  employ  in  a  complex 
organization.  The  EP-T  method  is  simpler  and  faster  to  use. 
It  lends  itself  to  those  applications  where  the  probability 
distribution  governing  some  process  is  symmetrical .  It  has 
the  added  advantage  of  supporting  traditional  Expected  Value 
analyses . 
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C.  AREAS  FOR  FURTHER  RESEARCH 

1.  Investigate  the  potential  dichotomies  between  the 
Program  Manager's  (PM)  perception  of  technical  risk  and 
those  of  the  Defense  Acquisition  Board  (DAB) . 

As  documented  in  this  thesis,  risk  perception  is  a 
specifically  individual  characteristic.  The  technical  risk 
inherent  in  a  new  system  acquisition  project  may  be 
perceived  differently  by  the  DAB,  the  PM,  and  various 
supporting  staffs. 

2.  Investigate  ordinal  technology  valuation  schemes 
applicable  for  all  military  systems. 

A  family  of  technology  valuation  schemes  using 
ordinal  scales  would  allow  for  explicit  quantitative 
comparison.  Such  a  device  would  allow  senior  management  to 
conduct  technologically  based  comparisons  between  U.S.  and 
potentially  hostile  forces. 

3.  Investigate  methods  of  conducting  Department  of 
Defense-wide  risk  portfolio  management. 

Methods  for  pooling  individual  quantitative  project 
technical  risk  assessment  information  into  a  central 
portfolio  should  be  explored.  Techniques  for  managing  the 
overall  DoD  technical  risk  position  should  be  investigated. 
This  could  provide  senior  decision  makers  with  a  more  useful 
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risk  indicator  of  DoD’s  exposure  to  technical  risk.  It 
would  also  provide  quantitative  data  on  which  to  base 
coherent  departmental  risk  seeking  or  risk  avoiding  actions. 
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APPENDIX  A 

TECHNOLOGY  REGRESSION  DATA 


AIRCRAFT 

IOC  YEAR 

TASCFORM 

CALCULATED 

COMPOSITE 

STATE-OF-THE 

INDEX 

ART 

F-89C 

50 

2.46 

4.35 

F-9F/H 

51 

4.19 

5.09 

F-89A 

51 

4.05 

5.09 

F-84F 

51 

5.13 

5.09 

F-86F 

51 

4.03 

5.09 

F-86D 

51 

3 . 68 

5.09 

F-2C 

51 

3.91 

5.09 

F-3A/B/C 

52 

9.02 

5.82 

F-IB/C/M 

52 

5.29 

5.82 

F~86H 

52 

5.68 

5.82 

F-IOOA/C 

52 

4.80 

5.82 

F-llA 

53 

5.80 

6.56 

A-3A/B 

53 

10.74 

6.56 

F-102A 

53 

9.71 

6.56 

F-6A 

53 

7.58 

6.56 

A-4A/B 

53 

3.93 

6.56 

F-IE 

54 

5.44 

7.30 

F-IOIA/B 

54 

13.55 

7.30 

F-IOOD 

54 

5.99 

7.30 

F-8A/B/C 

55 

8.40 

8.03 

A-IJ 

55 

3.34 

8.03 

F-9F  , 

55 

4.02 

8.03 

A-IE/G/H 

56 

3.34 

8.77 

F-104A/B 

56 

6.79 

8.77 

F-106A/B 

57 

13.05 

9.51 

F-105B/D 

57 

14.86 

9.51 

A-4C 

57 

5.45 

9.51 

F-4A/B 

59 

9.32 

10.98 

A-6A 

61 

13.83 

12.46 

A-4E/F 

61 

7.27 

12.46 

F-4C/D 

62 

10.07 

13.19 

P-3C 

65 

30.33 

15.41 

A-7A/B 

65 

12.10 

15.41 

F~111A 

65 

18.46 

15.41 

F-4E 

66 

13.96 

16.14 

F~111B 

66 

24.81 

16.14 

F-4J 

66 

13.39 

16.14 

A~7E 

68 

19.77 

17.62 

A-7D 

68 

16.17 

17.62 

F-lllD 

68 

24.39 

17.62 

S-3A 

69 

20.21 

18.35 

F-lllF 

70 

31.01 

19.09 

A-4M 

70 

8.52 

1:9.09 

A-6E 

70 

22.40 

19.09 
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APPENDIX  A 

TECHNOLOGY  REGRESSION  DATA 


AIRCRAFT 

IOC  YEAR 

TASCFORM 

CALCULATED 

COMPOSITE 

STATE-OF-THE 

INDEX 

ART 

F-14A 

71 

31.51 

19.83 

F-15A 

73 

16.14 

21.30 

A-lOA 

75 

12.12 

22.78 

F-16A 

78 

15.69 

24.99 

F/A-18A 

79 

25.42 

25.72 

'Ay 
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APPENDIX  A 

TECHNOLOGY  REGRESSION  DATA 


AIRCRAFT 

IOC  YEAR 

ADVANCE 

STANDARD 

RESIDUALS 

NORMAL 

PROBABILITY 

VALUES 

F-89C 

50 

-1.89 

-0.91478 

-1.10394 

F-9F/H 

51 

-0.90 

-1.05636 

-1.45013 

F-89A 

51 

-1.04 

0.80981 

1.01428 

F-84F 

51 

0.04 

-0.78958 

-1.01428 

F-86F 

51 

-1.06 

-2.08095 

-1 .84145 

F-86D 

51 

-1.41 

-0.51618 

-0.65252 

F-2C 

51 

-1.18 

-1.00697 

-1.20356 

F-3A/B/C 

52 

3.20 

0.26118 

0.47355 

F-IB/C/M 

52 

-0.53 

0.64542 

0.85598 

F-86H 

52 

-0.14 

-0.28700 

-0 . 30897 

F-IOOA/C 

52 

-1.02 

0.41603 

0.59076 

F-llA 

53 

-0.76 

-0 . 64585 

-0.85598 

A-3A/B 

53 

4.18 

-2.14163 

-2.23702 

F-102A 

53 

3.15 

-0.10807 

0 . 15261 

F-~6A 

53 

1.02 

-0.36498 

-0.41745 

A-4A/B 

53 

-2.63 

-0.23414 

-0.15264 

F~1E 

54 

-1.86 

0.61989 

0.78453 

F-IOIA/B 

54 

6.25 

-0.42871 

-0.59076 

F-IOOD 

54 

-1.31 

-0.53953 

-0.71693 

F-8A/B/C 

55 

0.37 

-0.32561 

-0.36267 

A-IJ 

55 

-4.69 

-0.60852 

-0.78453 

F-9F 

55 

-4.01 

0.19452 

0.41745 

A-IE/G/H 

56 

-5.43 

0.06684 

0.36267 

F-104A/B 

56 

-1.98 

-0.78286 

-0.93217 

F-106A/B 

57 

3.54 

-0.17934 

0.05069 

F-105B/D 

57 

5.35 

-0.15207 

0.10151 

A-4C 

57 

-4.06 

2.29787 

1.61546 

F~4A/B 

59 

-1.66 

-1.03021 

-1.31686 

A-6A 

61 

1.37 

-1.90119 

-1.61546 

A-4E/F 

61 

-5.19 

-0.06761 

0.20411 

F-4C/D 

62 

-3 . 12 

0.00460 

0.30897 

P-3C 

65 

14.93 

-0.27914 

-0.25618 

A-7A/B 

65 

-3.31 

-0.21065 

-0.10151 

F-lllA 

65 

-3.06 

-0.03196 

0.25618 

F~4E 

66 

-2.18 

-0.37473 

-0.47355 

F-lllB 

66 

8.67 

-0.20674 

-0.05069 

F~4J 

66 

-2.75 

-0.25810 

-0.20411 

A-7E 

68 

2.15 

-0.20370 

0.00000 

A-7D 

68 

-1.45 

1.17210 

1.20356 

F-lllD 

68 

6.77 

0.60925 

0.71693 

S-3A 

69 

1.86 

-0.38799 

-0.53122 

F-lllF 

70 

11.92 

1.03150 

1.10394 

A-4M 

70 

-10.57 

0.68122 

0.93217 

A-6E 

70 

3.31 

0.58843 

0.65252 

F-14A 

71 

11.68 

1.68065 

1.45013 
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TECHNOLOGY  REGRESSION  DATA 


AIRCRAFT 

IOC  YEAR 

ADVANCE 

STANDARD 

RESIDUALS 

F-15A 

73 

-5.16 

1.31825 

A-lOA 

75 

-10.66 

2.33663 

F-ieA 

78 

-9.30 

2.89203 

F/A-18A 

79 

-0.30 

0.35896 

NORMAL 

PROBABILITY 

VALUES 

1.31686 
1.84145 
2.2370 
0.5312 
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